A real-time speech synthesis method, device, apparatus and storage medium

Abstract

The application discloses a real-time speech synthesis method, device and equipment and a storage medium. The method comprises the following steps: performing request analysis on an obtained speech synthesis request to obtain a text to be synthesized, a reference audio identifier and a tracking identifier; performing text sentence processing on the text to be synthesized to generate at least one text sentence sequence, and obtaining reference audio according to the reference audio identifier; generating a marked prompt object conforming to a virtual large language model inference framework standard according to the corresponding text sentence sequence and the reference audio; calling a speech synthesis model generated based on the virtual large language model inference framework, performing acoustic processing on each marked prompt object based on pre-generated sampling parameters to obtain an acoustic unit sequence corresponding to each text sentence sequence; generating target synthesis audio according to each acoustic unit sequence, and generating a target access link according to the target synthesis audio and the tracking identifier, so that a speech synthesis demander can access the synthesis audio based on the target access link.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a real-time speech synthesis method, apparatus, device, and storage medium. Background Technology

[0002] With the rapid development of AI-based Text-to-Speech (TTS) technology, deep learning-based end-to-end speech synthesis systems have been widely applied in scenarios such as intelligent customer service, voice assistants, and audiobooks. Existing mainstream TTS systems typically adopt a two-stage architecture: the first stage uses an autoregressive model to convert text into acoustic features or discrete acoustic units, and the second stage uses a vocoder to restore the acoustic features into audio waveforms.

[0003] Existing speech synthesis technologies require step-by-step decoding of acoustic sequences, resulting in slow inference speed, low resource utilization, poor concurrency, and insufficient synthesis accuracy, failing to meet the needs of real-time voice interaction scenarios. Furthermore, in scenarios requiring timbre cloning, efficient inference for zero-sample timbre cloning scenarios cannot be achieved, further contributing to the low efficiency of speech synthesis. Summary of the Invention

[0004] This invention provides a real-time speech synthesis method, apparatus, device, and storage medium to improve the efficiency and accuracy of speech synthesis in scenarios with timbre cloning, so as to meet the needs of real-time speech synthesis interactive scenarios.

[0005] According to one aspect of the present invention, a real-time speech synthesis method is provided, the method comprising: In response to a speech synthesis request from a speech synthesis client, the speech synthesis request is parsed to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier. The text to be synthesized is processed by text segmentation to generate at least one text sentence sequence, and reference audio is obtained according to the reference audio identifier; Based on the corresponding text sentence sequence and the reference audio, generate a standard tag prompt object for the virtual large language model inference framework; The speech synthesis model generated based on the virtual large language model inference framework is invoked, and acoustic processing is performed on each of the marked prompt objects based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each of the text sentence sequences. Based on the sequence of each acoustic unit, a target synthesized audio is generated, and based on the target synthesized audio and the tracking identifier, a target access link is generated so that the speech synthesis requester can access the synthesized audio based on the target access link.

[0006] According to another aspect of the present invention, a real-time speech synthesis apparatus is provided, the apparatus comprising: The synthesis request response module is used to respond to the speech synthesis request from the speech synthesis requester, and to parse the speech synthesis request to obtain the text to be synthesized, the reference audio identifier, and the tracking identifier. The text segmentation processing module is used to perform text segmentation processing on the text to be synthesized, generate at least one text sentence sequence, and obtain reference audio according to the reference audio identifier; The prompt object generation module is used to generate a marked prompt object according to the standard of the virtual large language model inference framework based on the corresponding text sentence sequence and the reference audio. The acoustic processing module is used to call the speech synthesis model generated based on the virtual large language model inference framework, and perform acoustic processing on each of the marked prompt objects based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each of the text sentence sequences. An audio synthesis module is used to generate a target synthesized audio based on the sequence of each acoustic unit, and to generate a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the real-time speech synthesis method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the real-time speech synthesis method according to any embodiment of the present invention.

[0009] The technical solution of this invention involves parsing a speech synthesis request to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier; performing text segmentation on the text to be synthesized to generate at least one text sentence sequence, and obtaining a reference audio based on the reference audio identifier; generating a vLLM standard-compliant cue object based on the corresponding text sentence sequence and the reference audio; calling a speech synthesis model generated based on the vLLM framework to perform acoustic processing on each cue object to obtain a sequence of acoustic units; generating a target synthesized audio based on each acoustic unit sequence, and generating a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link. The above technical solution employs a speech synthesis model adapted to the vLLM framework for speech synthesis, and improves the model's speech synthesis efficiency based on the vLLM acceleration mechanism. In the zero-sample timbre cloning scenario, it generates vLLM-recognizable tag prompts based on speaker features, eliminating redundant conditional computations and overcoming the limitation of the vLLM framework natively supporting only discrete text tokens. This ensures that the speech synthesis model has the vLLM inference acceleration capability, balancing speech synthesis efficiency and accuracy in both real-time speech synthesis and timbre cloning scenarios, and meeting the needs of real-time speech synthesis interactive scenarios.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a real-time speech synthesis method provided in an embodiment of the present invention; Figure 2 A flowchart of a real-time speech synthesis method provided in an embodiment of the present invention; Figure 3 This is a schematic diagram of the structure of a real-time speech synthesis device provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device for the real-time speech synthesis method according to an embodiment of the present invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart of a real-time speech synthesis method provided in Embodiment 1 of the present invention. This embodiment is applicable to real-time speech synthesis interactions where timbre cloning is required. The method can be executed by a real-time speech synthesis device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. In response to the speech synthesis request from the speech synthesis requester, the speech synthesis request is parsed to obtain the text to be synthesized, the reference audio identifier, and the tracking identifier.

[0016] S120. Perform text segmentation on the text to be synthesized to generate at least one text sentence sequence, and obtain reference audio based on the reference audio identifier.

[0017] S130. Generate a marked prompt object according to the standard of the virtual large language model inference framework based on the corresponding text sentence sequence and reference audio.

[0018] S140. Call the speech synthesis model generated based on the virtual large language model inference framework, and perform acoustic processing on each marked prompt object based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each text sentence sequence.

[0019] S150. Generate target synthesized audio based on the sequence of each acoustic unit, and generate target access link based on the target synthesized audio and tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link.

[0020] The user requesting speech synthesis can be any third party with real-time speech synthesis needs, such as an e-commerce platform's intelligent voice customer service or an audiobook platform. Furthermore, the user can choose the desired voice tone, such as a "female official customer service voice," a "male official customer service voice," or an "anime character" voice. Users can select the voice tone according to their specific needs, using reference audio identifiers.

[0021] For example, a speech synthesis requester can initiate a speech synthesis request, which may include the text to be synthesized, a reference audio identifier, and a tracking identifier. The tracking identifier is used to characterize the uniqueness of the speech synthesis request; the reference audio identifier can be the identifier corresponding to the selected audio timbre by the speech synthesis requester, used to characterize the uniqueness of that audio timbre; the text to be synthesized can be the text that needs to be synthesized in real time. For example, in the above-mentioned e-commerce platform intelligent voice customer service scenario, e-commerce platform users access intelligent customer service via telephone. The platform converts the user's voice question into text and generates a customer service reply text based on the user's voice question text. This customer service reply text is the text to be synthesized. For example, the text to be synthesized generated by the e-commerce platform could be "Your order has been shipped and is expected to arrive in 3-5 days. If you have any questions, please call the customer service hotline 400-xxxx".

[0022] Each reference audio identifier has one and only one corresponding reference audio. The mapping relationship between the reference audio identifier and the reference audio can be pre-stored, and this mapping relationship can be represented by a timbre mapping table. Optionally, the timbre mapping table can store the mapping relationship between the reference audio identifier and the corresponding audio path of the reference audio. The reference audio can then be accessed based on the audio path, thereby reducing memory usage.

[0023] Upon receiving a speech synthesis request from a client, the system parses the request to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier. Parameter validation is then performed on the text to be synthesized, the reference audio identifier, and the tracking identifier, such as verifying the existence of the reference audio identifier in the timbre mapping table and confirming the tracking identifier's compliance. Optionally, a temporary storage directory can be created, with the directory name including the tracking identifier, to store subsequently generated intermediate and final audio data.

[0024] Based on the reference identifier, the audio path corresponding to the reference audio is obtained from the timbre mapping table, and the reference audio is obtained based on the audio path. The text to be synthesized is processed by text segmentation to generate at least one text sentence sequence, and each text sentence sequence contains at least one sentence segment.

[0025] In one optional embodiment, the text to be synthesized is processed into sentences to generate at least one text sentence sequence, including: performing text regularization cleaning on the text to be synthesized to obtain cleaned synthesized text; performing numerical range normalization processing on the cleaned synthesized text to obtain normalized synthesized text; performing phoneme conversion on the normalized synthesized text to obtain phoneme-converted text; and performing sentence segmentation on the phoneme-converted text to generate at least one text sentence sequence; the text sentence sequence includes at least one sentence word segmentation.

[0026] Specifically, regular expressions are used to remove meaningless special characters from the text to be synthesized, such as quotation marks, links within parentheses, and special symbols, resulting in clean synthesized text. The clean synthesized text undergoes numerical range normalization, converting hyphen-connected ranges into natural language expressions connected by the word "to". For example, the hyphen range "3-5 days" is replaced with "3 to 5 days", and slash-based units are converted to "per", such as "meters / second" being converted to "meters per second". Phoneme conversion is then performed on the normalized synthesized text. This can be achieved by calling a text front-end engine to perform word segmentation and phoneme conversion, converting Chinese characters into a sequence of tones in pinyin, resulting in phoneme-converted text.

[0027] The phoneme-converted text is segmented into sentences to generate at least one text sentence sequence. For example, continuing the previous example "Your order has been shipped and is expected to arrive in 3-5 days. If you have any questions, please call our customer service hotline 400-xxxx". After conversion, two independent text sentences are obtained: "Your order has been shipped and is expected to arrive in 3 to 5 days" and "If you have any questions, please call our customer service hotline 400xxxx". The text sentence sequence of each text sentence contains sentence segments. For example, the text sentence sequence of the first sentence can be ["your", "order", "shipped", "expected", "3 to 5 days", "arrived"].

[0028] Based on the corresponding text sentence sequence and reference audio, a standard labeled prompt object is generated for the virtual large language model inference framework. The virtual large language model framework can be the vLLM (Virtual Large Language Model) framework, a mainstream high-performance large language model inference and service framework. Its core goal is to make LLM inference fast and easy to use. Its core innovation is the PagedAttention algorithm, which manages the LLM's KV (Key-Value) cache like an operating system manages virtual memory, almost eliminating memory fragmentation, thereby significantly improving throughput, reducing GPU memory usage, and supporting ultra-long contexts and batch processing.

[0029] The speech synthesis model used in this embodiment can be pre-trained by relevant technical personnel, or an existing speech synthesis model can be used, such as an autoregressive acoustic model, a non-autoregressive acoustic model, and a parallel acoustic model. The vLLM framework and the speech synthesis model are pre-initialized.

[0030] Specifically, the process involves obtaining the speech synthesis model's configuration file and parsing its basic configuration parameters, such as network structure, weights, and paths. The vLLM asynchronous inference engine is initialized, configuring single-card tensor parallelism and setting the GPU (Graphics Processing Unit) memory utilization limit and data precision. The speech synthesis model is then registered in the vLLM model registry, and the adaptation interface is rewritten to enable the vLLM scheduler to recognize the speech synthesis model's input / output and attention calculation logic.

[0031] This embodiment injects the speech synthesis model into vLLM, enabling the speech synthesis model to adapt to vLLM. Specifically, it employs various techniques to ensure the speech synthesis model's compatibility with the vLLM framework. These include the implementation of a multimodal support interface, a multimodal processor, and corrected positional encoding calculation logic. The implementation of the multimodal support interface includes: implementing a method for obtaining multimodal embeddings, extracting audio embedding representations from input parameters, processing redundant dimensions, and returning a standardized embedding list; and implementing a method for obtaining input embeddings, merging text marker embeddings with audio embeddings, and using a multimodal embedding merging function to replace placeholder marker positions with consecutive embeddings.

[0032] In implementing the multimodal processor, a processing information class is predefined to declare support for audio modalities; a virtual input builder is predefined to provide virtual embeddings for pre-allocation of video memory; a data parser is defined to parse audio modal data and establish a mapping relationship between data keys and model parameters; a multimodal processor is defined, and a cue update rule is defined to map the embedding sequence length back to the number of placeholder markers.

[0033] The corrections to the position encoding calculation logic include: rewriting the input preparation method of the vLLM inference engine; calculating the cue marker offset for custom model types, where the offset is a negative value equal to the length of the cue marker sequence minus one; and accumulating the offset into the position index array to ensure that the position encoding is calculated from zero, thus avoiding position offset errors.

[0034] Based on the implementation of the aforementioned multimodal support interface, the multimodal processor, and the correction of the positional encoding calculation logic, the speech synthesis model becomes compatible with the vLLM framework. A marked cue object, conforming to the vLLM standard, is generated based on the corresponding text sentence sequence and reference audio, ensuring that subsequent vLLM interfaces can call this marked cue object.

[0035] In one optional embodiment, generating a labeled prompt object according to the standard of the Virtual Large Language Model Inference Framework based on the corresponding text sentence sequence and reference audio includes: generating a speaker conditional feature vector based on the reference audio; generating a target embedding sequence corresponding to the corresponding text sentence sequence based on the speaker conditional feature vector and the corresponding text sentence sequence; constructing a multimodal data dictionary based on the target embedding sequence; constructing a placeholder label sequence; and generating a labeled prompt object according to the standard of the Virtual Large Language Model Inference Framework based on the placeholder label sequence and the multimodal data dictionary.

[0036] It should be noted that users of speech synthesis services can select audio files with different timbres for speech synthesis according to their own needs. Therefore, in zero-sample timbre cloning scenarios, it is necessary to extract speaker features from the reference audio and fuse them with the features of the text to be synthesized. To further improve the accuracy of extracting speaker conditional feature vectors from the reference audio, this embodiment also provides a feature extraction method for the reference audio.

[0037] In one optional embodiment, generating a speaker conditional feature vector based on a reference audio includes: resampling the reference audio and then performing channel conversion to obtain mono audio; extracting Mel spectral features from the mono audio to obtain Mel spectral features; encoding the Mel spectral features to obtain a voice-coded feature vector; and resampling the voice-coded feature vector to obtain a speaker conditional feature vector.

[0038] Load the reference audio and resample it, for example, resample it to 24kHz. Then perform channel conversion to convert it into a mono signal. For example, take the left channel or average value from a multi-channel audio source to obtain a mono signal, resulting in mono audio. Specifically, the Mel spectrum can be calculated using Short-Time Fourier Transform (STFT). Core parameters can include window size (e.g., 1024), frame shift or jump length (e.g., 256), and the number of Mel filters (e.g., 100). For example, the reference audio is processed into frames, with 1024 samples per frame and a frame shift of 256. After windowing each frame, an FFT (Fast Fourier Transform) is performed to obtain the linear power spectrum. The linear power spectrum is multiplied by a 100-dimensional Mel filter bank to obtain the Mel filtered spectrum, and the logarithm is taken to obtain the Mel spectral characteristics.

[0039] The Mel-spectral features are encoded to obtain the voice-coded feature vector. Specifically, the Mel-spectral features can be input into the Conformer encoder to encode the Mel-spectral features, extract speaker features, and obtain the voice-coded feature vector, which includes features such as timbre and prosody. The Perceiver resampler is used to compress the voice-coded feature vector into a fixed-length conditional vector, and the feature sequence length is uniformized by masking, outputting the speaker conditional feature vector.

[0040] For any text sentence sequence, the speaker conditional feature vector and the corresponding text sentence sequence are fused in terms of features to generate a target embedding sequence corresponding to the corresponding text sentence sequence. In an optional embodiment, generating a target embedding sequence corresponding to the corresponding text sentence sequence based on the speaker conditional feature vector and the corresponding text sentence sequence includes: performing text embedding and positional encoding on the corresponding text sentence sequence to obtain a sentence text embedding vector; concatenating the corresponding sentence text embedding vector with the speaker conditional feature vector in the sequence dimension to obtain a conditionalized text vector; and adding a start marker to the conditionalized text vector to obtain the target embedding sequence corresponding to the corresponding text sentence sequence.

[0041] For any given text sentence sequence, the sequence is converted into a text embedding through an embedding layer. The corresponding vector is then extracted by position index and added element-wise to the text embedding, incorporating temporal position information to obtain the sentence text embedding vector corresponding to the text sentence sequence. This sentence text embedding vector is then concatenated with the speaker conditional feature vector along the sequence dimension to obtain a conditional text vector. The speaker conditional feature vector can be placed at the beginning of the sequence as a global timbre condition. Finally, an embedding vector with a start marker is added to the conditional text vector sequence to obtain the target embedding sequence corresponding to the text sentence sequence. Therefore, the generated target embedding sequence contains embedding vectors of speaker conditional features, sentence text features, and a start marker.

[0042] A multimodal data dictionary is constructed based on the target embedding sequence. The target embedding sequence is then used to construct the multimodal data dictionary, where the key is "audio modality" and the key is the target embedding sequence. A placeholder tag sequence is constructed, with a length that can be set to 1, and can be identified using special placeholder characters. Based on the placeholder tag sequence and the multimodal data dictionary, a vLLM-style tag cue object is generated. Specifically, the placeholder tag sequence and the multimodal data dictionary can be encapsulated into a standard vLLM cue object as input for subsequent model inference to obtain the vLLM-style tag cue object.

[0043] The speech synthesis model, generated based on the vLLM framework, is invoked. Based on pre-generated sampling parameters, acoustic processing is performed on each tagged cue object to obtain the acoustic unit sequences corresponding to each text sentence sequence. The sampling parameters can be pre-configured by relevant technical personnel according to actual needs. These parameters may include a temperature coefficient to control generation randomness; top-p kernel sampling (assuming a value of 0.9, meaning sampling is only performed from tokens with a cumulative probability of 90%); the number of top-k samples to limit the range of candidate tokens (e.g., set to 50); a repetition penalty coefficient to suppress repeated generation (e.g., set to 1.1); and a maximum number of generated tokens to avoid infinite generation (e.g., set to 1024).

[0044] It should be noted that the core acceleration mechanisms of vLLM can include: PagedAttention, ContinuousBatching, and CUDA Graph (Compute Unified Device Architecture Graph) optimization.

[0045] The core logic of pagination attention can be as follows: divide the KVCache into fixed physical pages (2048 tokens per page) and maintain the logical mapping through the page table; the KVCache for different requests does not need to be in contiguous video memory, which solves the fragmentation problem and improves the video memory utilization from 30% to over 80%.

[0046] The core logic of continuous batch processing can be: update the batch every time a token is generated, remove completed requests and add new requests, keeping the GPU at full load and improving throughput by 3-5 times compared to static batch processing. The core logic of CUDA Graph optimization can be: compile a fixed computation graph into a CUDA Graph, reducing CPU (Central Processing Unit) instruction issuance and kernel startup overhead, and reducing single-token latency by 20%-30%.

[0047] In one optional embodiment, a speech synthesis model generated based on a virtual large language model inference framework is invoked to perform acoustic processing on each marked cue object based on pre-generated sampling parameters, thereby obtaining acoustic unit sequences corresponding to each text sentence sequence. This includes: invoking a request structure based on the virtual large language model inference framework engine, passing in pre-generated sampling parameters and each marked cue object, so that the virtual large language model inference framework request scheduler can invoke the speech synthesis model to perform asynchronous acoustic task processing based on the marked cue objects, thereby obtaining acoustic unit sequences corresponding to each text sentence sequence.

[0048] The speech synthesis model configured with vLLM can effectively inherit vLLM's acceleration mechanism during acoustic unit sequence prediction, thereby improving the efficiency of model feature data processing. Specifically, it calls the asynchronous request interface `engine.add_request()` of the vLLM engine, passing in sampling parameters and each marked cue object, and can also pass in a tracking identifier to associate with the current speech synthesis request. After receiving the request, the vLLM scheduler adds it to the global waiting queue and schedules the speech synthesis model to perform feature processing to generate acoustic unit sequences. During the data processing of the speech synthesis model, vLLM's asynchronous task processing mechanism can ensure that each marked cue object is processed in parallel and continuously in batches, thereby improving concurrency throughput and eliminating batch waiting. Moreover, the KV caching mechanism based on PagedAttention paging can solve the problem of GPU memory fragmentation during model processing, thereby improving resource utilization. vLLM's fusion of CUDA operators and batch parallel computing mechanism can reduce the computation time of the model in a single step, thereby improving decoding speed. Therefore, the vLLM-based speech synthesis model performs feature processing based on the marked cue objects that vLLM can recognize, obtaining the acoustic unit sequence corresponding to each text sentence sequence.

[0049] For example, calling `engine.add_request()` submits a request, which enters the vLLM scheduling queue and is dynamically merged into a batch for execution with other concurrent requests. After waiting for asynchronous inference to complete, the token sequence is extracted from the returned result, and the last digit is removed. <eos>The end marker yields an acoustic unit token sequence of length M corresponding to each asynchronously processed text sentence sequence.

[0050] It should be noted that achieving compatibility between the custom speech synthesis model and the vLLM framework involves multiple core technologies, each employing different methodologies. These include the implementation of multimodal support interfaces, the implementation of multimodal processors, and the correction of positional encoding logic.

[0051] The implementation of the multimodal support interface mainly involves the `get_multimodal_embeddings` and `get_input_embeddings` methods. The `get_multimodal_embeddings` method extracts audio embeddings from the multimodal dictionary, processes the dimensions, and returns a normalized embedding list. The `get_input_embeddings` method locates placeholder positions and replaces one placeholder with 32 consecutive speaker embeddings, thus outputting a complete input embedding sequence.

[0052] The implementation of a multimodal processor mainly involves the `ProcessingInf`, `DummyInputBuilder`, `DataParser`, and `MultiModalProcessor` classes. The `ProcessingInf` class is primarily used to declare support for the audio modality, informing the scheduler of the model's capabilities. The `DummyInputBuilder` class is mainly used to construct virtual audio embeddings and text sequences for pre-allocation of video memory and performance profiling at startup. The `DataParser` class is mainly used to establish the mapping between multimodal data keys and model forward parameters, ensuring that the embeddings are correctly passed to the model. The `MultiModalProcessor` class is mainly used to define the mapping rules between placeholders and embedding lengths, with one placeholder corresponding to 32 embedding tokens, supporting correct allocation in the KVCache.

[0053] The correction of the positional encoding logic mainly includes rewriting the vLLM_prepare_inputs method to handle positional encoding separately for the custom speech synthesis model; calculating the positional offset: offset = -(cue marker sequence length - 1) to offset the positional offset caused by the speaker prefix; and accumulating the offset into the positional index array to ensure that the positional encoding of the text and acoustic parts starts from 0, which conforms to the original logic of the model.

[0054] Based on the sequence of acoustic units, a target synthesized audio is generated. Specifically, each acoustic feature can be transformed into a playable audio feature, and the resulting playable audio features can be concatenated to obtain the target synthesized audio.

[0055] In one optional embodiment, generating target synthesized audio based on each acoustic unit sequence includes: performing tensor format conversion on the corresponding acoustic unit sequence to determine the sequence length; generating continuous latent variable representation information based on the speaker conditional feature vector, the corresponding text sentence sequence and its corresponding acoustic unit sequence and sequence length; calling a vocoder to convert the continuous latent variable representation information into audio waveforms to obtain sentence audio waveforms corresponding to each text sentence sequence; and performing sentence-by-sentence audio concatenation on each sentence audio waveform to obtain the target synthesized audio.

[0056] For any given acoustic unit sequence, the acoustic unit token sequence is tensor-transformed into a tensor-formatted acoustic unit sequence, and the sequence length is determined. The forward propagation algorithm of the speech synthesis model is then invoked to generate continuous latent variable representations based on the speaker's conditional feature vector, the corresponding text sentence sequence, and the corresponding tensor-transformed acoustic unit sequence and sequence length. The scaling factor during generation can be set to 1024. Here, 1024 is the upsampling scaling factor, meaning each acoustic unit sequence corresponds to a latent variable with 1024 sampling points. The output is a continuous latent variable representation, i.e., Mel-spectral features, which serve as the input to the subsequent vocoder.

[0057] The vocoder converts the continuous latent variable representation information into audio waveforms, obtaining the sentence audio waveforms corresponding to each text sentence sequence. The vocoder can be a BigVGAN (Big Generative Adversarial Network) vocoder. BigVGAN's large kernel convolution can capture long-range temporal dependencies, improving naturalness; the anti-aliasing activation function suppresses upsampling high-frequency aliasing, avoiding noise. The amplitude of the sentence audio waveforms is limited to the range [-32767, 32767], and then converted to a 16-bit integer format.

[0058] The audio waveforms of each sentence are segmented and concatenated to obtain the target synthesized audio. Specifically, this can be achieved by concatenating the audio waveforms of all sentences in the order of the text along the time dimension to obtain the complete audio, i.e., the target synthesized audio. Optionally, core metrics of the target synthesized audio can also be calculated for subsequent analysis. For example, core metrics may include audio duration and implementation rate. Audio duration can be calculated by dividing the number of waveform sampling points by the sampling rate, and the implementation rate can be calculated by dividing the total inference time by the audio duration.

[0059] To further ensure the accuracy of the generated target synthesized audio, optionally, long silence segments in the target synthesized audio can be filtered out. The maximum number of consecutive silence markers can be set to 15, and the silence marker value to 52, retaining sequences where the silence markers do not exceed the threshold. The target synthesized audio is saved to a temporary directory and written to a WAV (WaveformAudio File Format) using a 24kHz sampling rate; the GPU memory cache is cleared to release the memory resources used during inference.

[0060] Based on the target synthesized audio and tracking identifier, a target access link is generated, allowing users to access the synthesized audio. Specifically, a storage client can be created using a key via an SDK (Software Development Kit) to establish a connection with the target storage bucket. The target synthesized audio file in WAV format is uploaded, named according to the tracking identifier for easy retrieval; for example, the audio file name could be "tts-service / {request_id}.wav", where "request_id" is the tracking identifier. An audio access URL is generated, and a JSON (JavaScript Object Notation) response is returned, containing the tracking identifier, audio download URL, audio duration, and failure reason fields.

[0061] The technical solution of this invention involves parsing a speech synthesis request to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier; performing text segmentation on the text to be synthesized to generate at least one text sentence sequence, and obtaining a reference audio based on the reference audio identifier; generating a vLLM standard-compliant cue object based on the corresponding text sentence sequence and the reference audio; calling a speech synthesis model generated based on the vLLM framework to perform acoustic processing on each cue object to obtain a sequence of acoustic units; generating a target synthesized audio based on each acoustic unit sequence, and generating a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link. The above technical solution employs a speech synthesis model adapted to the vLLM framework for speech synthesis, and improves the model's speech synthesis efficiency based on the vLLM acceleration mechanism. In the zero-sample timbre cloning scenario, it generates vLLM-recognizable tag prompts based on speaker features, eliminating redundant conditional computations and overcoming the limitation of the vLLM framework natively supporting only discrete text tokens. This ensures that the speech synthesis model has the vLLM inference acceleration capability, balancing speech synthesis efficiency and accuracy in both real-time speech synthesis and timbre cloning scenarios, and meeting the needs of real-time speech synthesis interactive scenarios.

[0062] Figure 2 This is a schematic diagram illustrating the flowchart of a real-time speech synthesis method provided in an embodiment of the present invention. Based on the above embodiments, this embodiment provides a preferred example. Figure 2 As shown, the method includes the following steps: S21. In response to the speech synthesis request from the speech synthesis requester, the speech synthesis request is parsed to obtain the text to be synthesized, the reference audio identifier, and the tracking identifier.

[0063] S22. Perform text segmentation on the text to be synthesized to generate at least one text sentence sequence, and obtain reference audio based on the reference audio identifier.

[0064] S23. Generate speaker conditional feature vectors based on the reference audio.

[0065] Specifically, the reference audio is resampled and then channel-converted to obtain mono audio; Mel-spectral features are extracted from the mono audio to obtain Mel-spectral features; the Mel-spectral features are encoded to obtain a voice-coded feature vector; and the voice-coded feature vector is resampled to obtain a speaker conditional feature vector.

[0066] S24. Based on the speaker's conditional feature vector and the corresponding text sentence sequence, generate the target embedding sequence corresponding to the corresponding text sentence sequence.

[0067] Specifically, the corresponding text sentence sequence is embedded and positionally encoded to obtain a sentence text embedding vector; the corresponding sentence text embedding vector is concatenated with the speaker conditional feature vector in the sequence dimension to obtain a conditional text vector; a start marker is added to the conditional text vector to obtain the target embedding sequence corresponding to the corresponding text sentence sequence.

[0068] S25. Construct a multimodal data dictionary based on the target embedding sequence.

[0069] S26. Construct a placeholder tag sequence and generate a tag prompt object according to the standard of the virtual large language model inference framework based on the placeholder tag sequence and the multimodal data dictionary.

[0070] S27. Call the request structure based on the virtual large language model inference framework engine, pass in the pre-generated sampling parameters and each marked prompt object, so that the virtual large language model inference framework request scheduler can call the speech synthesis model to perform asynchronous acoustic task processing based on the marked prompt objects, and obtain the acoustic unit sequence corresponding to each text sentence sequence respectively.

[0071] S28. Perform tensor format conversion on the corresponding acoustic unit sequence to determine the sequence length.

[0072] S29. Generate continuous latent variable representation information based on the speaker conditional feature vector, the corresponding text sentence sequence and its corresponding acoustic unit sequence and sequence length.

[0073] S210. Call the vocoder to convert the continuous latent variable representation information into audio waveforms, and obtain the sentence audio waveforms corresponding to each text sentence sequence.

[0074] S211. The audio waveforms of each sentence are segmented and spliced ​​together to obtain the target synthesized audio.

[0075] S212. Based on the target synthesized audio and tracking identifier, generate a target access link so that the speech synthesis requester can access the synthesized audio based on the target access link.

[0076] Figure 3 This is a schematic diagram of a real-time speech synthesis device provided in an embodiment of the present invention. The real-time speech synthesis device provided in this embodiment is applicable to real-time speech synthesis interactions where voice cloning is required. This real-time speech synthesis device can be implemented in hardware and / or software, such as... Figure 3 As shown, the device includes: a synthesis request response module 301, a text segmentation processing module 302, a prompt object generation module 303, an acoustic processing module 304, and an audio synthesis module 305. Among them, The synthesis request response module 301 is used to respond to the speech synthesis request from the speech synthesis requester, and to parse the speech synthesis request to obtain the text to be synthesized, the reference audio identifier, and the tracking identifier. The text segmentation processing module 302 is used to perform text segmentation processing on the text to be synthesized, generate at least one text sentence sequence, and obtain reference audio according to the reference audio identifier; The prompt object generation module 303 is used to generate a marked prompt object according to the standard of the virtual large language model inference framework based on the corresponding text sentence sequence and the reference audio. The acoustic processing module 304 is used to call the speech synthesis model generated based on the virtual large language model inference framework, and perform acoustic processing on each of the marked prompt objects based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each of the text sentence sequences. The audio synthesis module 305 is used to generate a target synthesized audio based on the sequence of each acoustic unit, and to generate a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link.

[0077] The technical solution of this invention involves parsing a speech synthesis request to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier; performing text segmentation on the text to be synthesized to generate at least one text sentence sequence, and obtaining a reference audio based on the reference audio identifier; generating a vLLM standard-compliant cue object based on the corresponding text sentence sequence and the reference audio; calling a speech synthesis model generated based on the vLLM framework to perform acoustic processing on each cue object to obtain a sequence of acoustic units; generating a target synthesized audio based on each acoustic unit sequence, and generating a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link. The above technical solution employs a speech synthesis model adapted to the vLLM framework for speech synthesis, and improves the model's speech synthesis efficiency based on the vLLM acceleration mechanism. In the zero-sample timbre cloning scenario, it generates vLLM-recognizable tag prompts based on speaker features, eliminating redundant conditional computations and overcoming the limitation of the vLLM framework natively supporting only discrete text tokens. This ensures that the speech synthesis model has the vLLM inference acceleration capability, balancing speech synthesis efficiency and accuracy in both real-time speech synthesis and timbre cloning scenarios, and meeting the needs of real-time speech synthesis interactive scenarios.

[0078] Optionally, the prompt object generation module 303 includes: The speaker conditional feature extraction unit is used to generate a speaker conditional feature vector based on the reference audio. The target embedding sequence generation unit is used to generate a target embedding sequence corresponding to the corresponding text sentence sequence based on the speaker conditional feature vector and the corresponding text sentence sequence; A multimodal data dictionary construction unit is used to construct a multimodal data dictionary based on the target embedding sequence; The tag hint object generation unit is used to construct a placeholder tag sequence and generate a tag hint object according to the virtual large language model inference framework standard based on the placeholder tag sequence and the multimodal data dictionary.

[0079] Optional, speaker conditional feature extraction unit, specifically used for: The reference audio is resampled and then channel-switched to obtain mono audio. Mel spectrum features are extracted from the mono audio signal to obtain the Mel spectrum features. The Mel spectrum features are encoded to obtain the human voice encoded feature vector; The human voice coding feature vector is resampled to obtain the speaker conditional feature vector.

[0080] Optionally, the target embedding sequence generation unit is specifically used for: The corresponding text sentence sequence is embedded and positionally encoded to obtain the sentence text embedding vector; The corresponding sentence text embedding vector is concatenated with the speaker conditional feature vector in the sequence dimension to obtain the conditional text vector; Add a start marker to the conditionalized text vector to obtain the target embedding sequence corresponding to the text sentence sequence.

[0081] Optional, acoustic processing module 304, specifically used for: The request structure based on the virtual large language model inference framework engine is invoked, and pre-generated sampling parameters and each marked cue object are passed in, so that the request scheduler based on the virtual large language model inference framework can call the speech synthesis model to perform asynchronous acoustic task processing based on the marked cue objects, and obtain the acoustic unit sequence corresponding to each of the text sentence sequences.

[0082] Optional, the audio synthesis module 305 is specifically used for: Perform tensor format conversion on the corresponding acoustic unit sequence to determine the sequence length; Based on the speaker conditional feature vector, the corresponding text sentence sequence and its corresponding acoustic unit sequence, and the sequence length, continuous latent variable representation information is generated; The vocoder is invoked to convert the continuous latent variable representation information into an audio waveform, thereby obtaining the sentence audio waveforms corresponding to each of the text sentence sequences; The audio waveforms of each sentence are segmented and spliced ​​together to obtain the target synthesized audio.

[0083] Optional, the text segmentation processing module 302 is specifically used for: The text to be synthesized is cleaned using text regular expressions to obtain cleaned synthesized text; The cleaned and synthesized text is subjected to numerical range normalization processing to obtain normalized synthesized text; The standardized synthesized text is subjected to phoneme conversion to obtain phoneme-converted text; The phoneme-converted text is segmented into sentences to generate at least one text sentence sequence; the text sentence sequence includes at least one sentence segmentation.

[0084] The real-time speech synthesis device provided in the embodiments of the present invention can execute the real-time speech synthesis method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0085] Figure 4 A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0086] like Figure 4 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0087] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0088] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as real-time speech synthesis methods.

[0089] In some embodiments, the real-time speech synthesis method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the real-time speech synthesis method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the real-time speech synthesis method by any other suitable means (e.g., by means of firmware).

[0090] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0091] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0092] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0093] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0094] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0095] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0096] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0097] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.< / eos>

Claims

1. A real-time speech synthesis method, characterized in that, include: In response to a speech synthesis request from a speech synthesis client, the speech synthesis request is parsed to obtain the text to be synthesized, a reference audio identifier, and a tracking identifier. The text to be synthesized is processed by text segmentation to generate at least one text sentence sequence, and reference audio is obtained according to the reference audio identifier; Based on the corresponding text sentence sequence and the reference audio, generate a standard tag prompt object for the virtual large language model inference framework; The speech synthesis model generated based on the virtual large language model inference framework is invoked, and acoustic processing is performed on each of the marked prompt objects based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each of the text sentence sequences. Based on the sequence of each acoustic unit, a target synthesized audio is generated, and based on the target synthesized audio and the tracking identifier, a target access link is generated so that the speech synthesis requester can access the synthesized audio based on the target access link.

2. The method according to claim 1, characterized in that, The step of generating a labeled prompt object according to the corresponding text sentence sequence and the reference audio, based on the standard of the virtual large language model inference framework, includes: Based on the reference audio, a speaker conditional feature vector is generated; Based on the speaker conditional feature vector and the corresponding text sentence sequence, generate the target embedding sequence corresponding to the corresponding text sentence sequence; Construct a multimodal data dictionary based on the target embedding sequence; Construct a placeholder tag sequence, and generate a tag hint object according to the standard of the virtual large language model inference framework based on the placeholder tag sequence and the multimodal data dictionary.

3. The method according to claim 2, characterized in that, The step of generating a speaker conditional feature vector based on the reference audio includes: The reference audio is resampled and then channel-switched to obtain mono audio. Mel spectrum features are extracted from the mono audio signal to obtain the Mel spectrum features. The Mel spectrum features are encoded to obtain the human voice encoded feature vector; The human voice coding feature vector is resampled to obtain the speaker conditional feature vector.

4. The method according to claim 2, characterized in that, The step of generating the target embedding sequence corresponding to the corresponding text sentence sequence based on the speaker conditional feature vector and the corresponding text sentence sequence includes: The corresponding text sentence sequence is embedded and positionally encoded to obtain the sentence text embedding vector; The corresponding sentence text embedding vector is concatenated with the speaker conditional feature vector in the sequence dimension to obtain the conditional text vector; Add a start marker to the conditionalized text vector to obtain the target embedding sequence corresponding to the text sentence sequence.

5. The method according to claim 1, characterized in that, The process of calling the speech synthesis model generated based on the virtual large language model inference framework, and performing acoustic processing on each of the marked prompt objects based on pre-generated sampling parameters to obtain the acoustic unit sequences corresponding to each of the text sentence sequences, includes: The request structure based on the virtual large language model inference framework engine is invoked, and pre-generated sampling parameters and each marked cue object are passed in, so that the request scheduler based on the virtual large language model inference framework can call the speech synthesis model to perform asynchronous acoustic task processing based on the marked cue objects, and obtain the acoustic unit sequence corresponding to each of the text sentence sequences.

6. The method according to claim 2, characterized in that, The step of generating the target synthesized audio based on the sequence of each acoustic unit includes: Perform tensor format conversion on the corresponding acoustic unit sequence to determine the sequence length; Based on the speaker conditional feature vector, the corresponding text sentence sequence and its corresponding acoustic unit sequence, and the sequence length, continuous latent variable representation information is generated; The vocoder is invoked to convert the continuous latent variable representation information into an audio waveform, thereby obtaining the sentence audio waveforms corresponding to each of the text sentence sequences; The audio waveforms of each sentence are segmented and spliced ​​together to obtain the target synthesized audio.

7. The method according to claim 1, characterized in that, The step of performing text segmentation on the text to be synthesized to generate at least one text sentence sequence includes: The text to be synthesized is cleaned using text regular expressions to obtain cleaned synthesized text; The cleaned and synthesized text is subjected to numerical range normalization processing to obtain normalized synthesized text; The standardized synthesized text is subjected to phoneme conversion to obtain phoneme-converted text; The phoneme-converted text is segmented into sentences to generate at least one text sentence sequence; the text sentence sequence includes at least one sentence segmentation.

8. A real-time speech synthesis device, characterized in that, include: The synthesis request response module is used to respond to the speech synthesis request from the speech synthesis requester, and to parse the speech synthesis request to obtain the text to be synthesized, the reference audio identifier, and the tracking identifier. The text segmentation processing module is used to perform text segmentation processing on the text to be synthesized, generate at least one text sentence sequence, and obtain reference audio according to the reference audio identifier; The prompt object generation module is used to generate a marked prompt object according to the standard of the virtual large language model inference framework based on the corresponding text sentence sequence and the reference audio. The acoustic processing module is used to call the speech synthesis model generated based on the virtual large language model inference framework, and perform acoustic processing on each of the marked prompt objects based on the pre-generated sampling parameters to obtain the acoustic unit sequence corresponding to each of the text sentence sequences. An audio synthesis module is used to generate a target synthesized audio based on the sequence of each acoustic unit, and to generate a target access link based on the target synthesized audio and the tracking identifier, so that the speech synthesis requester can access the synthesized audio based on the target access link.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the real-time speech synthesis method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the real-time speech synthesis method according to any one of claims 1-7.

Images