Speech synthesis and simultaneous interpretation processing method, device and product

Abstract

The invention provides a speech synthesis and simultaneous interpretation processing method, device and product. The method comprises the following steps: acquiring a text feature block which is to be subjected to speech synthesis and comprises a plurality of text feature frames; for each text feature frame in the plurality of text feature frames, based on a text feature block corresponding to the current text feature frame and a spliced semantic coding feature corresponding to a historical text feature frame before the current text feature frame, determining a semantic coding feature corresponding to the current text feature frame; splicing the semantic coding feature corresponding to the current text feature frame and the spliced semantic coding feature corresponding to the historical text feature frame to obtain a spliced semantic coding feature corresponding to the current text feature frame; and if the preset output condition comprises that speech synthesis is performed according to the text feature block and the current text feature frame is the last text feature frame in the plurality of text feature frames, performing speech synthesis according to the spliced semantic coding feature corresponding to the current text feature frame. According to the invention, the rhythm naturalness of speech synthesis can be improved.

Description

Technical Field

[0001] This application relates to the field of speech processing, and in particular to a speech synthesis, simultaneous interpretation processing method, device and product. Background Technology

[0002] Text-to-Speech (TTS) technology, as an important branch of artificial intelligence, has been widely used in many fields in recent years, greatly improving user experience and interaction efficiency.

[0003] In intelligent speech-to-speech translation (S2ST) systems, real-time cross-language communication at near-human levels can be achieved through the cascading of automatic speech recognition (ASR) systems, machine translation (MT) systems, and speech synthesis systems.

[0004] However, the speech synthesized by existing speech synthesis technology is usually unnatural in rhythm and has a strong mechanical feel, resulting in low fluency of speech synthesis. Summary of the Invention

[0005] Based on the aforementioned technological status, this application provides a speech synthesis and simultaneous interpretation processing method, device, and product that can improve the rhythmic naturalness of synthesized speech, reduce the mechanical feel, and thus enhance the fluency of speech synthesis.

[0006] To achieve the above-mentioned technical objectives, this application proposes the following technical solution: According to a first aspect of the embodiments of this application, a speech synthesis method is provided, comprising: acquiring a text feature block to be synthesized, the text feature block including a plurality of text feature frames; for each text feature frame in the plurality of text feature frames, determining a semantic coding feature corresponding to the current text feature frame based on the text feature block corresponding to the current text feature frame and the concatenated semantic coding features corresponding to the historical text feature frames preceding the current text feature frame; concatenating the semantic coding feature corresponding to the current text feature frame and the concatenated semantic coding features corresponding to the historical text feature frames to obtain the concatenated semantic coding feature corresponding to the current text feature frame; if a preset output condition includes speech synthesis based on the text feature block, and the current text feature frame is the last text feature frame in the plurality of text feature frames, then speech synthesis is performed based on the concatenated semantic coding feature corresponding to the current text feature frame.

[0007] According to a second aspect of the embodiments of this application, a simultaneous interpretation processing method is provided, comprising: acquiring a speech to be processed from a target speaker, wherein the language of the speech to be processed is a source language; performing speech recognition on the speech to be processed to obtain a speech recognition result; translating the speech recognition result into text in the target language, and determining text feature blocks to be synthesized based on the text in the target language; performing speech synthesis processing on the text feature blocks using the speech synthesis method as described in the first aspect to obtain a speech synthesis result; and outputting the speech synthesis result.

[0008] According to a third aspect of the present application, an electronic device is provided, including a memory and a processor; the memory is connected to the processor and is used to store a program; the processor is used to implement the speech synthesis method as described in the first aspect by running the program in the memory.

[0009] According to a fourth aspect of the embodiments of this application, a computer program product is provided, including computer program instructions that, when executed by a processor, cause the processor to perform: the speech synthesis method as described in the first aspect.

[0010] This application provides a speech synthesis method, apparatus, device, and product. The method acquires a text feature block to be synthesized, which includes multiple text feature frames. For each text feature frame, based on the text feature block corresponding to the current text feature frame and the concatenated semantic coding features corresponding to historical text feature frames preceding the current text feature frame, a semantic coding feature corresponding to the current text feature frame is determined. The semantic coding feature corresponding to the current text feature frame and the concatenated semantic coding features corresponding to historical text feature frames are concatenated to obtain the concatenated semantic coding feature corresponding to the current text feature frame. If the preset output conditions include speech synthesis based on text feature blocks, and the current text feature frame is the last text feature frame among multiple text feature frames, then speech synthesis is performed based on the concatenated semantic coding feature corresponding to the current text feature frame. This technical solution, by introducing information from complete text feature blocks and combining it with historical concatenated semantic coding features, enables the semantic coding of each text frame to fully integrate contextual semantics, sentence structure, and tone intention, improving semantic integrity and intonation naturalness, optimizing prosodic boundary prediction, and reducing abruptness in speech concatenation. Meanwhile, by using historical semantic encoding as a memory mechanism, a smooth transition of speech features between frames is achieved, ensuring temporal coherence and expression consistency, and significantly improving the naturalness, fluency and semantic coherence of synthesized speech. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0012] Figure 1 This is a schematic diagram of a simultaneous speech interpretation system in related technologies.

[0013] Figure 2 This is a flowchart of a speech synthesis method provided in an embodiment of this application.

[0014] Figure 3 This is a schematic diagram illustrating the principle of a training process for an acoustic model provided in an embodiment of this application.

[0015] Figure 4 A flowchart illustrating the joint training of the acoustic model and vocoder provided in this application embodiment.

[0016] Figure 5 This is a schematic diagram of the structure of the vocoder provided in an embodiment of this application.

[0017] Figure 6 A flowchart illustrating the training process of the semantic coding model provided in this application embodiment.

[0018] Figure 7 This is a schematic diagram illustrating the principle of the semantic coding model training process provided in the embodiments of this application.

[0019] Figure 8 This is a schematic diagram of the autoregressive inference process provided in an embodiment of this application.

[0020] Figure 9 This is a flowchart for training a semantic coding model to control speech rate, provided as an embodiment of this application.

[0021] Figure 10 This is a schematic diagram illustrating the principle of training a semantic coding model for speech rate control, as provided in an embodiment of this application.

[0022] Figure 11 This is a schematic diagram illustrating speech rate control training of an acoustic model, as provided in an embodiment of this application.

[0023] Figure 12 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application.

[0024] Figure 13 This is a schematic diagram of the structure of a simultaneous interpretation processing device provided in an embodiment of this application.

[0025] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] The technical solutions provided in this application can be applied, by way of example, to hardware devices such as processors, electronic devices, and servers (including cloud servers), or packaged as software programs and run. When the hardware device executes the processing procedure of the technical solutions in this application, or when the aforementioned software program is run, the target task can be automatically split and the application programming interfaces required by the task can be automatically invoked to achieve the purpose of the target task. This application only provides illustrative descriptions of the specific processing procedure of the technical solutions in this application and does not limit the specific implementation form of the technical solutions in this application. Any technical implementation form that can execute the processing procedure of the technical solutions in this application can be adopted by this application.

[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] Before introducing the solution proposed in this application, the relevant technologies will first be introduced: Figure 1 This is a schematic diagram of a simultaneous speech interpretation system in related technologies. For example... Figure 1 As shown, the simultaneous interpretation system includes: a speech recognition system 101, a machine translation model 102, and a speech synthesis system 103; Among them, the speech recognition system 101 is used to perform speech recognition on the speaker's speech of the source language category and obtain the speech recognition result of the source language category.

[0029] Machine translation model 102 is used to translate speech recognition results of the source language category into text data of the target language category.

[0030] The speech synthesis system 103 is used to synthesize speech based on text data of a target language category to obtain synthesized speech of the target language category.

[0031] Taking simultaneous Chinese-English interpretation as an example, assuming the source language is Chinese and the target language is English, after the Chinese speech is collected, it needs to be recognized by the speech recognition system 101 to obtain the Chinese speech recognition result, i.e., Chinese text data; then, the machine translation model 102 translates the Chinese text data to obtain the English text data; finally, the speech synthesis system 103 synthesizes the English text data to obtain the English speech data.

[0032] As the final output link in the entire technology chain of intelligent simultaneous speech interpretation, TTS not only undertakes the crucial task of converting translated text into speech and completing the "listening-translation-speaking" closed loop, but is also the core hub of user experience. Its output quality directly determines the system's naturalness, credibility, and real-time performance. If speech synthesis suffers from problems such as a mechanical feel, stiff intonation, excessive latency, or broken splicing, even if the front-end recognition and translation are accurate, users will have a negative perception of the overall system as "unprofessional" or "untrustworthy." To improve TTS performance, current TTS tasks mainly include the following core steps: Step 1: Input text preprocessing. Receive incomplete sentences from machine translation and segment them based on semantic boundaries (such as commas, conjunctions, and syntactic structures).

[0033] Step 2: Prosodic Parameter Generation. Using the TTS front-end model, the segmented text from Step 1 is converted into the input format required by the acoustic model, which typically includes information such as phonemes, prosodic features, and intonation stress.

[0034] Step 3: Acoustic Modeling and Incremental Synthesis. Using acoustic models, such as FastSpeech 2, parallel prediction of phonemes to Mel spectra is achieved.

[0035] Step 4: Speech Waveform Generation. Using a neural network vocoder, such as WaveRNN or LPCNet, the Mel spectrum is converted into a high-quality speech waveform, completing the final speech output.

[0036] In addition, the following optional enhancement modules can be introduced to meet specific needs: Step 5 (optional): Multilingual mixing processing, including language boundary detection and cross-language prosodic smoothing, to ensure a smooth audio transition when switching languages ​​and avoid abrupt changes.

[0037] Step Six: Dynamic post-processing, such as forced calibration of terminology pronunciation, using a pronunciation dictionary or rule engine to calibrate the pronunciation of key words such as professional terms, personal names, or place names to improve accuracy.

[0038] Step 7: Resource scheduling and optimization, such as multi-path concurrent synthesis, to improve the synthesis speed of the system.

[0039] Current simultaneous interpretation solutions are mostly based on lightweight TTS models, and the prosodic performance of their synthesized sentences depends on the semantic completeness of the machine-translated output sentences. Furthermore, the recognition and translation stages rely on complete sentence input, and the inherent limitations of sequence-to-sequence (Seq2Seq) methods make true simultaneous listening and translation difficult to achieve. This problem leads to the continuous accumulation of latency in the process, making it difficult to overcome the upper limit of the overall response time. In addition, the models have strict requirements on the input format, requiring necessary conversion of the text to be synthesized, making the overall system process quite lengthy. In this cascaded system, errors at any stage will be propagated and accumulated, ultimately causing immeasurable impact on the output quality.

[0040] There are already numerous examples to reference regarding targeted improvements to system performance. For instance, addressing the issue that traditional TTS cannot preserve the emotional tone of the source speech, resulting in a mechanical output, EmoSpeech proposed CLIP-based speech-emotion alignment to achieve zero-sample emotion transfer and improve the prosodic emotional expression of the synthesized speech. Meanwhile, to address resource consumption and real-time bottlenecks, NVIDIA SPADE employs dynamic computation scheduling, predicting computation paths based on sentence complexity, using lightweight models for simple sentences, and triggering full-model inference for complex sentences. These strategies all address specific weaknesses and optimize individual aspects of the system.

[0041] However, the aforementioned solutions mostly focus on single dimensions such as "naturalness" or "latency." In intelligent simultaneous interpretation systems, TTS essentially requires balancing the equilateral triangle relationship formed by "naturalness, latency, and robustness." Since the operation of the entire system depends on this balance, an imbalance in any one aspect can introduce unpredictable defects. Currently, most solutions only optimize one direction, and the overall performance ceiling of the system ultimately depends on its "weakest link." Therefore, optimizing only a single aspect is unlikely to achieve a significant improvement in overall performance.

[0042] Therefore, the main technical problems that existing simultaneous interpretation speech synthesis modules need to address are as follows: First, in order to control the delay time of the preceding steps and ensure the real-time performance of the speech output, the segmentation precision of the speech units is often not very high. This leads to incomplete semantics in the synthesized segments, resulting in problems such as unnatural rhythm, abrupt splicing, and obvious mechanical feel, which affects the user's intuitive experience of the entire system. Second, if the segmentation precision of the speech units is increased to ensure semantic integrity, it will lead to insufficient real-time performance, high latency, and the difference between the inference time and the speech playback time of longer segments will lead to a decrease in computing power utilization during multi-channel concurrent inference. Third, the existing system only supports a single speech rate style, which is not flexible enough when dealing with diverse simultaneous interpretation application scenarios.

[0043] In summary, the relevant technologies suffer from the problem that rhythmic naturalness and real-time performance cannot be satisfied simultaneously, and the speech rate cannot be adjusted.

[0044] In addition, when dealing with meaningless text that has repeated fields or no clear semantic boundaries, the related technologies may encounter problems such as the inability to segment the text, resulting in excessively long text accumulation.

[0045] In view of this, the embodiments of this application aim to provide a speech synthesis, simultaneous interpretation processing method, device, and product. In the technical solution of this method, the target text segment to be synthesized is first acquired and divided into various text feature blocks, each containing multiple text feature frames. For each text feature frame, the semantic encoding features of the current frame are derived by utilizing not only the overall contextual information of its text feature block but also combining previously generated historical semantic encoding features. When a preset output timing is reached, the current semantic encoding and historical semantic encoding are fused to form a context-coherent concatenated semantic encoding feature. Subsequently, based on this fused feature, corresponding acoustic features are generated, and finally, high-quality synthesized speech is output. These will be described in detail in the following embodiments.

[0046] Exemplary methods Figure 2 This is a flowchart illustrating a speech synthesis method provided in an embodiment of this application. Figure 2 As shown, the speech synthesis method provided in this embodiment includes steps S201-S204: S201. Obtain the text feature block to be used for speech synthesis, which includes multiple text feature frames.

[0047] In simultaneous speech interpretation scenarios, text feature blocks can be obtained by extracting linguistic features and segmenting text fragments output from the translation model. The specific implementation process includes the following steps a1-a4: Step a1: Obtain the target text segment to be used for speech synthesis.

[0048] The target text fragment can be obtained from preceding modules, such as machine translation models, and used as the input source for speech synthesis.

[0049] In some examples, the length of the target text segment is determined by the closing marker in the text output by the preceding module. For example, using the end of a complete sentence as the dividing unit ensures semantic integrity and the natural flow of the synthesized speech.

[0050] Step a2: Extract linguistic features from the target text fragment to obtain the text feature segment.

[0051] In this embodiment, the text feature segment can be understood as the linguistic features corresponding to the target text fragment. The linguistic features include at least one of the following: phoneme information, paralinguistic sequences, and speaker timbre information.

[0052] Phoneme information represents the basic acoustic units of speech, and can be understood as phonetic symbols. It can be obtained through a preset phoneme list, which includes the mapping relationship between characters and their corresponding phonemes.

[0053] Paralinguistic sequences refer to non-lexical components that do not carry semantic meaning but exist in natural speech flow, such as breathing sounds, pauses, and modal particles (e.g., "um," "ah"). Introducing paralinguistic sequences helps improve the naturalness and auditory comfort of synthesized speech.

[0054] Speaker timbre information refers to the distinctive characteristics of a speaker's voice, including vocal timbre and voice timbre. Vocal timbre is determined by the tuning of the vocal organs, such as the degree of mouth opening and tongue position, typically manifested in the differences in pronunciation of different vowels (such as / a / and / e / ). Voice timbre is mainly determined by the physiological characteristics of the vocal cords (such as length and thickness), and has significant individual distinctiveness, such as the differences between male and female voices, and between adult and child voices.

[0055] Step a3: Segment the text feature segments according to the preset segmentation rules to obtain each text feature block.

[0056] Since the target text segment may be quite long, direct speech synthesis can easily lead to processing delays and affect the real-time performance of the speech output. Therefore, it is necessary to segment the text feature segments into individual text feature blocks according to preset segmentation rules in order to balance synthesis efficiency and speech quality.

[0057] If the segmentation granularity is too large, it will lead to incomplete semantics in the synthesized speech, unnatural rhythm, abrupt splicing, and a strong mechanical feel, affecting the overall fluency; if the segmentation granularity is too small, it will be impossible to guarantee real-time performance.

[0058] To ensure both the overall fluency and real-time performance of the synthesized speech, this embodiment sets segmentation rules based on preset numbers of syllables, characters, words, or punctuation marks. For example, each text feature block can be set to contain no more than 50 Chinese characters, no more than 15 words, or forced segmentation points can be set with periods and semicolons, thereby generating text feature units with controllable length and relatively complete semantics.

[0059] Compared to simply segmenting by a fixed length, this embodiment introduces linguistic features such as syllables, words, and punctuation marks as segmentation criteria. This results in more uniform segmentation lengths and finer granularity, making the segmentation points closer to the rhythm and grammatical structure of natural language. For example, segmenting at the end of a period preserves complete semantic units and avoids semantic fragmentation; segmenting at commas or phrase boundaries helps preserve reasonable intonation pauses. This makes the resulting speech, after independent synthesis of each text feature block, more natural in terms of rhythm, intonation, and semantic coherence, reducing mechanicalness and abruptness. By limiting the length of each text feature block, the processing load of a single speech synthesis task can be effectively controlled, response latency shortened, and waiting caused by processing excessively long texts avoided. This ensures that speech can be output quickly and continuously, meeting the needs of real-time interaction.

[0060] Step a4: Determine each text feature block in each text feature block as the text feature block to be used for speech synthesis.

[0061] That is, after obtaining each text feature block, the following steps S202 can be performed for each text feature block in each text feature block.

[0062] This embodiment describes first extracting linguistic features from the target text segment and then segmenting it into blocks. However, in other embodiments, the target text segment can be segmented according to a preset segmentation rule to obtain individual text blocks; then, linguistic features can be extracted from each text block to obtain the corresponding text feature blocks; finally, each text feature block is determined as the text feature block to be used for speech synthesis.

[0063] After obtaining the text feature blocks, the corresponding text feature frames can be determined based on the speech frame units identified in the speech recognition stage. That is, the linguistic features corresponding to the speech frame units identified in the speech recognition stage within each segmented text block are the text feature frames.

[0064] S202. For each text feature frame in multiple text feature frames, determine the semantic coding features corresponding to the current text feature frame based on the text feature block corresponding to the current text feature frame and the spliced ​​historical semantic coding features corresponding to the historical text feature frames before the current text feature frame.

[0065] This step employs an autoregressive inference mechanism, decoding multiple text feature frames in each text feature block frame by frame to sequentially generate semantic coding features corresponding to each text feature frame, namely VQ Code (Vector Quantized Code). VQ Code is a discretized representation of a continuous semantic space, which can be regarded as an encoded representation of text at a deep semantic level, and is used for subsequent acoustic modeling and speech synthesis.

[0066] In some embodiments, this step can be implemented using a semantic encoding model. The semantic encoding model is trained on a large language model. Therefore, when decoding any text feature frame, the model not only focuses on the current frame itself but also "sees" the complete text feature block to which it belongs (i.e., a segmented short text unit). This allows it to obtain global information such as the overall semantics, sentence structure, and tone of the segment, thereby improving semantic integrity and intonation naturalness, optimizing internal prosodic boundary prediction, reducing abrupt splicing caused by contextual fragmentation, and enhancing the understanding of pronouns and elliptical sentences. This ultimately improves the naturalness, prosodic rationality, and semantic integrity of the speech.

[0067] In the autoregressive decoding process, the semantic encoding of the current frame relies not only on the complete text feature block but also on the previously generated historical semantic encoding features (i.e., the VQ Code output of the previous frame). By passing the historical semantic encoding as "memory" to the next frame, features such as semantics, intonation, and timbre can smoothly transition between frames, avoiding jumps or breaks between frames and improving auditory fluency. Combining the historical spliced ​​speech encoding features for inference endows the model with "memory," enabling it to maintain the temporal coherence and expressive consistency of the output during frame-by-frame generation, thus achieving a streaming synthesis effect.

[0068] Among them, "historical splicing semantic coding features" refers to the semantic coding features that have been successfully decoded and generated before the current text feature frame for the current text feature block. Its specific composition will be further explained in step S103 in conjunction with the output mechanism.

[0069] S203. Concatenate the semantic coding features corresponding to the current text feature frame and the concatenated semantic coding features corresponding to the historical text feature frames to obtain the concatenated semantic coding features corresponding to the current text feature frame.

[0070] Specifically, the semantic coding features corresponding to the current text feature frame are sequentially concatenated with the concatenated semantic coding features corresponding to all previously generated historical text feature frames to form a continuous concatenated semantic coding feature sequence for use by the subsequent acoustic decoding module.

[0071] For example, suppose the text feature block obtained in step S201 includes 10 text feature frames: When decoding the first text feature frame, there are no historical text feature frames, and the concatenated semantic encoding features corresponding to the historical text feature frames are empty; When decoding the second text feature frame, the semantic coding features of the first text feature frame are used as the concatenated semantic coding features of its corresponding historical text feature frames for decoding the second frame. After obtaining the semantic coding features of the second frame, the semantic coding features of the first two frames are concatenated to obtain the concatenated semantic coding features corresponding to the second frame. When decoding the third text feature frame, the concatenated semantic coding feature of the second frame is used as the concatenated semantic coding feature of its corresponding historical text feature frame for decoding the third frame. After obtaining the semantic coding feature of the third frame, the semantic coding feature of the third frame is concatenated with the concatenated semantic coding feature corresponding to the second frame to obtain the concatenated semantic coding feature corresponding to the third frame. ... After decoding the 10th text feature frame, the semantic coding features of the 10th frame are concatenated with the concatenated semantic coding features corresponding to the 9th frame to obtain the concatenated semantic coding features corresponding to the 10th frame, which is the concatenated semantic coding features corresponding to the entire text feature block.

[0072] The "concatenated semantic coding features corresponding to historical text feature frames prior to the current text feature frame" mentioned in step S202 refers to the semantic coding features that have been generated and concatenated during the processing of the current text feature block. These features reflect the historical semantic information of the decoded text feature frames and are used to support the context modeling and semantic coherence of the current text feature frame.

[0073] S204. If the preset output conditions include speech synthesis based on text feature blocks, and the current text feature frame is the last text feature frame among multiple text feature frames, then speech synthesis is performed based on the concatenated semantic coding features corresponding to the current text feature frame.

[0074] In this embodiment, text feature blocks can be used as the basic unit of speech synthesis. That is, after decoding a text feature block and obtaining the concatenated semantic coding features of the text feature block, it is output and used for subsequent acoustic feature prediction and speech synthesis.

[0075] In some embodiments, text feature segments can also be used as the basic units for speech synthesis. That is, after decoding the last text feature block in a text feature segment, the semantic coding features of all text feature blocks are output and used for subsequent acoustic feature prediction and speech synthesis. Specifically, this includes: if the current text feature frame is the last text feature frame among multiple text feature frames, then the concatenated semantic coding feature corresponding to the current text feature frame is determined to be the semantic coding feature block corresponding to the text feature block; if the preset output conditions include speech synthesis based on text feature segments, and the current text feature block is the last text feature block among multiple text feature blocks, then speech synthesis is performed based on the semantic coding feature blocks corresponding to each text feature block.

[0076] This embodiment, by introducing complete text feature block information, fully integrates linguistic features such as global semantics, sentence structure, and tone intention from the context when predicting semantic encoding for each frame. This effectively improves semantic integrity and intonation naturalness, optimizes the prediction accuracy of prosodic boundaries, reduces abruptness in speech splicing caused by contextual fragmentation, and enhances the understanding of complex linguistic phenomena such as pronoun reference and elliptical sentences. This significantly improves the naturalness, prosodic rationality, and semantic coherence of synthesized speech. Simultaneously, by continuously transmitting previously generated historical spliced ​​semantic encoding features as "memory" information, a smooth transition of speech features such as semantics, intonation, and timbre between frames is achieved, avoiding jumps or breaks between consecutive frames and ensuring the temporal coherence and expression consistency of speech generation. Furthermore, by segmenting the text segments output by the preceding module and using text feature blocks as basic units for semantic encoding prediction, both the integrity of contextual modeling and processing efficiency are considered, helping to meet the real-time requirements of speech synthesis.

[0077] In some embodiments, steps S202 and S203 in the above embodiments can be implemented by a semantic coding model. The semantic coding model is trained on a Large Language Model (LLM) based on training text feature blocks and their corresponding training labels.

[0078] The training text feature block includes each training text feature frame. The semantic coding model to be trained predicts the semantic coding features corresponding to each training text feature frame based on the autoregressive inference mechanism. Based on the difference between the predicted semantic coding features and the real semantic coding features, the semantic coding model to be trained is converged to obtain the trained semantic coding model.

[0079] The autoregressive inference mechanism refers to using the training text feature blocks corresponding to the current training text feature frame and the historical predicted semantic coding features corresponding to previous training text feature frames as training samples to infer the predicted semantic coding features corresponding to that training text feature frame. The training process of the semantic coding model is described below with reference to the accompanying diagram: Figure 3 A flowchart illustrating the training process of the semantic coding model provided in this application embodiment. For example... Figure 3 As shown, the training process of the semantic coding model includes the following steps S301-S304: S301. Obtain the training text feature blocks and their corresponding training labels. The training labels include real semantic encoding features, and the training text feature blocks include multiple training text feature frames.

[0080] In some embodiments, obtaining training text feature blocks includes the following steps b1-b3: Step b1: Obtain training text fragments.

[0081] In this embodiment, the training text fragment includes multiple training statements, and the lengths of these training statements follow a normal distribution. Step b1, obtaining the training text fragment, includes the following steps b11 and b12: Step b11: Obtain the original training text, which includes multiple training statements.

[0082] The original training text is generated through speech recognition of language samples. Speech samples can be obtained from open source data and other channels, with preference given to data containing rich audio information, covering as many speakers as possible, and encompassing various language categories, age groups, and speech speed styles to ensure the diversity of data distribution.

[0083] In this step, the original training text is segmented by identifying terminating punctuation marks (such as periods, question marks, exclamation marks, etc.) to obtain multiple training sentences.

[0084] Step b12: If the number of first-type training statements is greater than the preset number and the number of second-type training statements is less than the preset number, the first-type training statements are concatenated to make the sentence length of the concatenated training statements greater than or equal to the preset sentence length, thereby obtaining the training text fragment corresponding to the original training text.

[0085] The first type of training sentences consists of sentences with a length less than the preset sentence length, while the second type consists of sentences with a length greater than or equal to the preset sentence length.

[0086] In this step, by statistically analyzing the sentence length distribution of the training sentences, the first type of training sentences (i.e., short sentences with a length less than a preset threshold) are spliced ​​together to optimize the overall sentence length distribution, making it more uniform and closer to a normal distribution.

[0087] The first type of training statements can be understood as short sentences, and the second type as long sentences. By statistically analyzing the sentence length distribution of multiple training statements, if it is found that the proportion of short sentences is too high and the proportion of long sentences is too low, the overall sentence length distribution can be adjusted by splicing short sentences to make it approach a normal distribution. This strategy helps to balance the length distribution of the training data and improve the model's coherence and expressiveness in document-level text synthesis.

[0088] Step b2: Extract linguistic features from the training text fragments to obtain the training text feature segments.

[0089] In this embodiment, the training text feature segment can be understood as the linguistic features corresponding to the training text fragment. The linguistic features include at least one of the following: phoneme information, paralinguistic sequences, and speaker timbre information.

[0090] For the definitions of phoneme information, paralinguistic sequence, and speaker timbre information, please refer to the introduction in step a2, which will not be repeated here.

[0091] Step b3: Segment the training text feature segments according to the preset segmentation rules to obtain each training text feature block.

[0092] In this embodiment, segmentation rules can be set according to a preset number of syllables, a preset number of characters, a preset number of words, or a preset number of punctuation marks. For example, each text feature block can be set to contain no more than 50 Chinese characters, or no more than 15 words, or forced segmentation points can be set with periods and semicolons, thereby generating training text feature blocks with controllable length and relatively complete semantics.

[0093] Step b4: Determine each training text feature block in each training text feature block as a training text feature block for training.

[0094] By limiting the length of each training text feature block, the processing load of a single speech synthesis task can be effectively controlled, the response latency can be shortened, and the waiting caused by processing excessively long text can be avoided, thereby ensuring that speech can be output quickly and continuously to meet the needs of real-time interaction.

[0095] The true semantic coding features in the training labels can be extracted using a trained semantic coding feature extraction model.

[0096] This semantic encoding feature extraction model is typically based on large-scale, high-quality text corpora and trained using supervised or self-supervised learning methods. It has the ability to map input text into a dense representation in a high-dimensional semantic vector space. During feature extraction, each training sentence or text fragment is input into the model, which processes it layer by layer through its deep neural network structure (such as the Transformer encoder), capturing lexical, syntactic, and semantic information in the text. Finally, it outputs the corresponding semantic encoding feature vector, which serves as the true representation of the text in the semantic space, i.e., the "true semantic encoding feature".

[0097] These real semantic coding features serve as training labels to guide the learning process of the semantic coding model to be trained, enabling it to approximate the ideal semantic representation capability during training.

[0098] S302. For each training text feature frame in each training text feature frame, the semantic coding model to be trained predicts the current semantic coding feature based on the predicted concatenated semantic coding features corresponding to the training text feature blocks and the historical training text feature frames before the current training text feature frame, and determines the predicted semantic coding feature corresponding to the current training text feature frame.

[0099] Figure 4 This is a schematic diagram illustrating the principle of the training process of the semantic coding model provided in the embodiments of this application. Figure 4 As shown, the semantic coding model includes an embedding layer 401, a large language model (LLM) 402, and a fully connected layer (FC) 403; wherein, the embedding layer 401 includes a speaker timbre information embedding layer 4011 and a phoneme sequence embedding layer 4012.

[0100] The speaker timbre information embedding layer 4011, namely the first embedding layer, is used to embed the speaker timbre information corresponding to the training text feature block to obtain the speaker timbre embedding representation spk_emb corresponding to the training text feature block. The phoneme sequence embedding layer 4012 is used to embed the phoneme sequence and sub-language sequence corresponding to the training text feature block respectively, to obtain the phoneme sequence and sub-language sequence embedding representation corresponding to the training text feature block, i.e., ph_emb.

[0101] Next, the speaker timbre embedding representation spk_emb, phoneme sequence and sub-language sequence embedding representation ph_emb corresponding to the training text feature block, and the predicted semantic coding feature vq_emb[1:] corresponding to the historical training text feature frame are concatenated to obtain the combined vector cat[ph_emb, spk_emb, vq_emb[1:]].

[0102] Subsequently, the combined vector is input into the LLM, enabling the LLM to predict the predicted semantic encoding features corresponding to the current training text feature frame.

[0103] S303. Concatenate the predicted semantic coding features corresponding to the current training text feature frame and the predicted concatenated semantic coding features corresponding to the historical training text feature frames before the current training text feature frame to obtain the predicted concatenated semantic coding features corresponding to the current training text feature frame.

[0104] Specifically, by concatenating the predicted semantic coding features of the current frame with the previously generated historical predicted semantic coding features in chronological order, a continuously accumulating and context-coherent sequence of predicted semantic coding features is constructed to enhance the model's ability to model long-term semantic dependencies.

[0105] The above embodiments describe preset output conditions including speech synthesis based on text feature blocks. In some other embodiments, the preset output conditions can be extended to speech synthesis based on longer-granularity text feature segments to adapt to synthesis needs in different scenarios and improve the fluency and naturalness of speech output. The following, with reference to the accompanying drawings, describes the specific implementation process of semantic encoding prediction by the semantic encoding model when the preset output conditions include speech synthesis based on text feature segments: Figure 5 This is a schematic diagram illustrating the autoregressive inference process for speech synthesis based on text feature segments, provided as an embodiment of this application. Figure 5 As shown, suppose the target text fragment is: It looks like it's going to rain this afternoon. Would you like me to remind you to bring an umbrella? After chunking, the resulting text chunks are: chunk1 is "It looks like", chunk2 is "it's going to", and the last chunk is "umbrella". Then, features are extracted from each chunk to obtain its corresponding feature blocks, i.e., individual text feature blocks. A semantic coding model (GPT) then predicts the corresponding VQ code based on the extracted features of each chunk. For example, the VQ code for chunk1 is predicted; then, combined with the VQ code of chunk1, the VQ code for chunk2 is predicted, and so on, until the VQ code of the last chunk is obtained.

[0106] During this inference process, when the inference is completed for the last training text block in the training text segment, all predicted VQ codes are output.

[0107] S304. If the preset output conditions include speech synthesis based on text feature blocks, and the current training text feature frame is the last training text feature frame among multiple training text feature frames, then based on the difference between the predicted concatenated semantic coding features corresponding to the current training text feature frame and the real semantic coding features, the semantic coding model to be trained is converged to obtain the trained semantic coding model.

[0108] A loss function is calculated based on the difference between the predicted concatenated semantic coding features and the corresponding true semantic coding features. The parameters of the semantic coding model under training are then updated using the backpropagation algorithm, gradually reducing the error between the predicted and true semantic coding features. This process is repeated iteratively until the model performance converges, resulting in a trained semantic coding model. This process typically employs a standard supervised learning framework, utilizing a gradient optimizer (such as Adam) to tune the parameters, ensuring that the model accurately maps the input text to the target semantic feature space.

[0109] The convergence conditions may include: the loss function value is less than a preset value, the number of training iterations reaches a preset number, or the model accuracy reaches a preset accuracy.

[0110] In some embodiments, to improve the naturalness and personalization of speech synthesis and make the generated synthesized speech as close as possible to the original speaker's expression habits in terms of speech rate features, the training labels not only include real semantic coding features, but can also further introduce the real speaker's speech rate as an additional supervision signal. This real speaker's speech rate is usually calculated by analyzing the time alignment information (such as phoneme or word-level time boundaries) of the corresponding speech samples, for example, by quantifying the speech rate as the number of utterances per unit time (such as phonemes / second, words / second, or phrases / second).

[0111] Based on this, the input training text feature blocks are used as input samples for the model, while the corresponding real speaker's speech rate is used as the training target to guide the training process of speech rate control in the semantic coding model. Specifically, further fine-tuning can be performed on the semantic coding model that has already been trained; or, speech rate labels can be introduced in the initial training stage of the model, and the semantic coding model to be trained can be directly trained end-to-end, so that it can learn semantic representations while also having the ability to model speaker speech rate features. The training process is described below with reference to the attached figures: Figure 6 This is a flowchart illustrating the speech rate control training of a semantic coding model, provided as an embodiment of this application. Figure 6 As shown, the training steps for the semantic coding model also include the following steps S601-S604: S601. Predict the speaker's speech rate corresponding to the feature block of the training text using the semantic coding model to be trained, and obtain the predicted speaker's speech rate.

[0112] In this step, the training text feature blocks are input into the semantic coding model to be trained in the current state. While generating semantic coding features, the model also outputs a prediction of the speaker's speech rate. The semantic coding model extracts high-level semantic representations through its internal network structure (e.g., a Transformer-based encoder) and introduces one or more dedicated output heads for regressing and predicting the speaker's speech rate. The prediction result is represented numerically, such as the number of phonemes per second, the number of characters per second, or a normalized speech rate embedding vector. This predicted value is the "predicted speaker speech rate," serving as the basis for subsequent calculations of the speech rate loss.

[0113] In some examples, the model can infer the rhythm pattern of a speaker's speech from the content, structure, semantics, and style features of the training text feature blocks. For example, long sentences and complex sentences usually have a slower speech rate with more pauses; short sentences and colloquial expressions tend to be pronounced quickly with continuous linking. The model analyzes structural information such as punctuation, dependency syntax, and phrase boundaries to infer the overall rhythm tendency.

[0114] In addition, paragraphs with a high density of verbs and information (such as scientific and technical expository texts) may have a faster speech rate; lyrical and emphatic words (such as "ah", "really?") are often accompanied by a slower speech rate to enhance expressiveness; mood keywords (excitement, nervousness, sadness) also affect the speech rate pattern. Therefore, the model can also identify these patterns through semantic embedding.

[0115] Moreover, function words such as "de", "le", "ne" often appear near the break points in the speech flow. The model can also learn to associate certain word combinations with specific speech rate changes, for example, there is often a slight deceleration + pause between listed items. [[ID=ID=7]]

[0116] In addition, using the self-attention mechanism or recurrent structure, the model can perceive the position and role of the current word in the whole sentence; for example, the beginning and end of a sentence are often slower, and the middle part is faster; the end of an interrogative sentence has an ascending tone and is drawn out, etc.

[0117] In the training stage, by statistically calculating the actual pronunciation duration of the training text feature blocks and calculating the average speech rate (e.g., number of syllables / second); using this speech rate value as the training label, that is, the true speaker's speech rate, the semantic encoding model learns the mapping relationship from the corresponding text feature blocks; after multiple rounds of training, the model can predict a reasonable expected speech rate given the text feature blocks.

[0118] Figure 7 Schematic diagram of the principle of training the semantic encoding model for speech rate control provided by the embodiments of this application. As Figure 7 shown, the semantic encoding model includes an embedding layer 401, a large language model (LLM) 402, and a fully connected layer (FC) 403; <​​​​The speaker rate information encoding layer 4013, also known as the style encoding layer, is used to encode the speaker rate information corresponding to the feature blocks of the training text, thereby obtaining the speaker rate encoding information style_emb corresponding to the feature blocks of the training text.

[0119] Next, the speaker timbre embedding representation spk_emb, speaker speech rate encoding information style_emb, phoneme sequence and sub-language sequence embedding representation ph_emb, and the predicted semantic encoding feature vq_emb[1:] corresponding to the training text feature blocks are concatenated to obtain the combined vector cat[style_emb, ph_emb, spk_emb, vq_emb[1:]].

[0120] Subsequently, the combined vector is input into the LLM, enabling the LLM to predict the speaker's speech rate corresponding to the current training text feature frame.

[0121] S602. Based on the difference between the predicted concatenated semantic coding features and the real semantic coding features, determine the semantic coding feature prediction loss.

[0122] This step measures the model's accuracy in semantic representation learning. "Predicted concatenated semantic coding features" refer to the sequence of semantic coding features output by the model to be trained for the current training samples, which may have undergone time-step alignment or frame-level concatenation to match the target dimension. "True semantic coding features" originate from a pre-trained high-precision semantic coding feature extraction model, obtained through forward inference on the corresponding speech or text content, representing the ideal semantic representation. The difference between the two can be calculated using standard distance metrics such as mean squared error (MSE), cosine similarity loss, or L1 loss, thus yielding the semantic coding feature prediction loss. This loss reflects the model's bias in capturing deep semantic information and is one of the core supervision signals for optimizing the model's semantic understanding ability.

[0123] S603. Based on the difference between the predicted speaker's speech rate and the actual speaker's speech rate, determine the speaker's speech rate prediction loss.

[0124] The actual speaker's speaking speed can be obtained by statistically analyzing the duration of the phoneme sequence corresponding to the original training text segment.

[0125] To enhance the model's ability to model speaker rhythm, a speech rate consistency constraint is introduced. The "real speaker speech rate" is the actual rate of articulation statistically calculated based on the time alignment results of the original speech signal and text (such as phoneme boundaries obtained through forced alignment techniques), possessing high realism and individual discriminability. The "predicted speaker speech rate" obtained in step S901 is compared with this real value, and a regression loss function (such as mean squared error (MSE) or smoothed L1 loss) is used to calculate the deviation between the two, yielding the speaker speech rate prediction loss. This loss term prompts the model to implicitly learn and reflect the speech rate habits of different speakers when generating semantic codes, thereby improving the performance of synthesized speech in terms of rhythm, pauses, and naturalness of speech flow.

[0126] S604. Based on the sum of the semantic coding feature prediction loss and the speaker's speech rate prediction loss, determine the comprehensive loss, and converge the semantic coding model to be trained according to the comprehensive loss to obtain the trained semantic coding model.

[0127] To achieve joint optimization of semantic accuracy and speech rate consistency, the semantic encoding feature prediction loss in step S602 and the speaker speech rate prediction loss in step S603 are weighted and summed to form a comprehensive loss function. Each loss term can be assigned a corresponding balancing coefficient to adjust the optimization priority between different tasks and prevent any single loss from dominating the overall training process. Subsequently, the gradient is calculated using the backpropagation algorithm based on this comprehensive loss, and the model parameters are updated using an optimizer (such as Adam or SGD). The entire training process iterates continuously until the preset convergence conditions are met (e.g., the validation set loss no longer decreases, the maximum number of training epochs is reached, or the gradient change tends to stabilize). Finally, a trained semantic encoding model that accurately models text semantics and reflects speaker speech rate characteristics is obtained, providing strong support for subsequent high-quality, personalized speech synthesis.

[0128] This embodiment introduces speech rate information as a supervisory signal, enabling the model to implicitly capture rhythmic and flow characteristics relevant to a specific speaker during the encoding process. Once the model is trained, in practical applications, it can predict semantic encoding representations consistent with the original speaker's speech rate tendency based solely on the contextual information and semantic context implied in the input text feature blocks. This drives subsequent acoustic models to generate synthesized speech that is more natural, realistic, and consistent with the individual's expressive style in terms of speech rate and rhythm. This method helps enhance the personalized expressive capabilities and contextual adaptability of speech synthesis systems, and is particularly suitable for application scenarios requiring high-fidelity reproduction of the speaker's language style.

[0129] In some embodiments, to improve the performance of the semantic coding model, a step-by-step training strategy can be adopted. First, training text feature blocks containing phoneme sequences and paralinguistic sequences are used as training samples, and real semantic coding features are used as training labels to perform base pre-training on the LLM to obtain a pre-trained semantic coding model. Then, training text feature blocks containing phoneme sequences, paralinguistic sequences, and speaker timbre information are used to perform a second-stage fine-tuning training on the pre-trained semantic coding model to obtain an intermediate semantic coding model. Finally, by increasing the proportion of target speaker data in the training samples to form new training samples, the intermediate semantic coding model is fine-tuned to conform to the target speaker's expression habits.

[0130] After obtaining the concatenated semantic coding features corresponding to the current text feature frame, speech synthesis can be performed based on them. In some embodiments, speech synthesis based on the concatenated semantic coding features corresponding to the current text feature frame includes: determining the acoustic features corresponding to the concatenated semantic coding features through an acoustic model; and determining the synthesized speech corresponding to the semantic coding features based on the acoustic features; wherein the acoustic model is trained by training the semantic coding features, linguistic features, and real acoustic features corresponding to the text; the linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences.

[0131] Figure 8 This is a schematic diagram illustrating the principle of a training process for an acoustic model provided in an embodiment of this application. Figure 8 As shown, the acoustic model includes an encoder 801 and a decoder 802; wherein, the encoder 801 is used to encode phoneme encoding features based on phoneme sequences and sub-language sequences.

[0132] Decoder 802 is used to perform joint decoding based on phoneme coding features and fusion features to obtain acoustic features. These acoustic features can be Mel-spectrogram features, or multimodal acoustic parameters such as linear spectrum, F0 fundamental frequency, and energy, which are used by the subsequent vocoder to synthesize speech waveforms.

[0133] Among them, the speaker timbre features and semantic coding features after the speaker timbre information (Speaker id) has been processed by embedding are concatenated to obtain the concatenated features (cat[spk_emb, vq_emb[1:]]); and the concatenated features are then used as the fusion features.

[0134] Specifically, the training steps for the acoustic model include: using the semantic encoding features and linguistic features corresponding to the training text as training samples, and using real acoustic features as training labels. The following section, in conjunction with the appendix... Figure 8 Introducing the training process of the acoustic model: The training process of the acoustic model includes the following steps c1-c2: Step c1: Obtain the predicted acoustic features using the acoustic model to be trained based on the semantic encoding features and linguistic features corresponding to the training text.

[0135] Continue reading Figure 8 Step c1 includes the following steps c11-c14: Step c11: By embedding the speaker timbre information corresponding to the training text, speaker timbre features are obtained.

[0136] Step c12: Fuse the speaker's timbre features and the semantic coding features corresponding to the training text to obtain the training fused features.

[0137] Step c13: Encode the phoneme sequence and sub-language sequence corresponding to the training text using encoder 801 to obtain the phoneme encoding features corresponding to the training text.

[0138] Step c14: The decoder 802 performs joint decoding based on the phoneme encoding features and training fusion features corresponding to the training text to obtain the predicted acoustic features.

[0139] Step c2: Determine the acoustic feature prediction loss based on the difference between the predicted acoustic features and the actual acoustic features, and converge the acoustic model to be trained based on the acoustic feature prediction loss to obtain the trained acoustic model.

[0140] The loss function is calculated based on the difference between predicted and true acoustic features. The parameters of the acoustic model being trained are then updated using backpropagation to gradually reduce the error between the predicted and true acoustic features. This process is repeated iteratively until the model performance converges, resulting in a trained acoustic model. This process typically employs a standard supervised learning framework, utilizing a gradient optimizer (such as Adam) for parameter tuning. Convergence conditions may include: the loss function value being less than a preset value, the number of training iterations reaching a preset number, or the model accuracy reaching a preset accuracy.

[0141] In some embodiments, in order to improve the performance of the acoustic model, a step-by-step training strategy can also be adopted. That is, the acoustic model to be trained is first trained using the semantic coding features, linguistic features and real acoustic features corresponding to the first training text to obtain an intermediate acoustic model; the intermediate acoustic model is then trained using the semantic coding features, linguistic features and real acoustic features corresponding to the second training text to obtain the acoustic model.

[0142] The training text for the first stage can be obtained using text data corresponding to high-quality speech data. Compared to the training text for the first stage, the training text for the second stage can increase the proportion of speaker data with the target timbre, thus enabling fine-tuning training with the target speaker.

[0143] After obtaining the trained acoustic model, the concatenated semantic encoded features can be input into the acoustic model to obtain the corresponding acoustic features. Then, the synthesized speech corresponding to the semantic encoded features is determined based on the acoustic features.

[0144] This embodiment adopts a phased prediction architecture of "text → semantic encoding → acoustic features", which not only reduces the complexity of model learning, but also enhances the controllability and interpretability of the speech generation process, making the modeling of each speech attribute more refined, and further improving the overall quality of synthesized speech.

[0145] In some embodiments, determining the synthesized speech corresponding to the semantic coding features based on acoustic features includes: determining the synthesized speech corresponding to the semantic coding features based on acoustic features using a vocoder; the vocoder is obtained through joint training with an acoustic model, such as... Figure 9 As shown, the joint training includes the following steps S901-S903: S901. Based on the semantic coding features and linguistic features corresponding to the training text, the acoustic features are predicted using an acoustic model to obtain the predicted acoustic features.

[0146] The acoustic model used in this step is the one trained through the aforementioned training steps. By inputting the semantic encoding features and linguistic features corresponding to the training text into the trained acoustic model, the corresponding predicted acoustic features can be obtained.

[0147] For the specific implementation process of step S901, please refer to the above. Figure 8 The processing procedures for encoders and decoders in the acoustic model will not be elaborated here.

[0148] It should be noted that the training text here can be the same as the training text of the acoustic model, or it can be completely different, or partially the same.

[0149] S902. The synthesized speech is predicted by the vocoder based on the predicted acoustic features.

[0150] By further inputting the predicted acoustic features into the vocoder to be trained for speech synthesis prediction, the corresponding predicted synthesized speech can be obtained.

[0151] In some examples, the vocoder can adopt the Bigvgan model. The main contribution of the Bigvgan model is in the generator, while the discriminator follows the structure of the classic Gan vocoder, including: multi-period discriminator (MPD), multi-scale discriminator (MSD), and multi-resolution discriminator (MRD).

[0152] Figure 10 This is a schematic diagram of the generator structure in the Bigvgan model provided in an embodiment of this application. Figure 10 As shown, the generators in the Bigvgan model include: One-dimensional convolutional layers (Conv1d) are used to perform one-dimensional convolution processing on acoustic features, extract local temporal patterns and enhance feature representation capabilities. They are suitable for modeling short-range dependencies of speech signals on the time axis.

[0153] The one-dimensional transposed convolutional layer (TransposedConv 1d) is used to upsample the acoustic features after one-dimensional convolution, gradually improving the temporal resolution of the feature sequence and achieving fine reconstruction from low-frequency acoustic parameters to high-sampling-rate waveforms.

[0154] AMP (Anti-aliased Multi-periodicity Composition) is the core technology in the BigVGAN model for high-quality waveform generation. By jointly modeling multiple periodic structures in the speech signal (such as fundamental frequency, harmonics, timbre, etc.) and introducing an anti-aliasing mechanism, it effectively suppresses artifacts caused by bandwidth folding during upsampling, significantly improving the reproduction capability and auditory fidelity of high-frequency components. This module includes the following key components: An upsampled low-pass filter is used to filter out high-frequency noise above the Nyquist frequency during the upsampling process of acoustic features after one-dimensional transposed convolution, preventing aliasing distortion and ensuring the spectral purity of the generated waveform.

[0155] The one-dimensional snake activation layer (Snake 1d) is used to model highly periodic signals (such as human voice and instrument waveforms) based on the acoustic features after upsampling. It can effectively capture the harmonic structure in speech and promote the recovery of complex waveform details.

[0156] A downsampled low-pass filter is used to filter out high-frequency interference before downsampling the features output by a one-dimensional serpentine activation layer in a multi-scale discriminator or feature compression path. This avoids aliasing errors introduced during the downsampling process and maintains feature consistency and temporal stability.

[0157] Dilated convolutional layers (Dilatedconv 1d) are used to process the output of downsampled low-pass filters. By expanding the receptive field without reducing temporal resolution, they capture long-range contextual dependencies. They are particularly suitable for modeling prosody and segmental structures across multiple frames in speech, enhancing the model's context awareness without increasing network depth.

[0158] Finally, the output of the dilated convolutional layer is passed through an activation layer (usually linear activation or tanh activation) to obtain the audio waveform of the predicted synthesized speech.

[0159] S903. Based on the difference between the predicted synthesized speech and the real synthesized speech, determine the speech synthesis prediction loss, and converge the vocoder based on the speech synthesis prediction loss to obtain the trained vocoder.

[0160] In this step, the acoustic model is set to a frozen state. This means that after determining the speech synthesis loss, the vocoder model parameters are updated only based on the speech synthesis loss, without updating the acoustic model parameters.

[0161] In this embodiment, a joint training approach is adopted, using acoustic features directly predicted by the acoustic model as input to optimize the vocoder in a way that more closely resembles real-world application scenarios. Compared to traditional methods that directly extract acoustic features (such as Mel spectrograms, F0, etc.) from training text as training samples, acoustic features generated by a pre-trained acoustic model can more accurately simulate the statistical characteristics and distribution of real speech signals. Since the feature distribution predicted by the acoustic model is closer to the input feature distribution of the vocoder in real-world applications, this helps to significantly reduce problems such as muffled voices (no sound or abnormally weak sound) and electronic voices (metallic or other unnatural timbre) caused by feature mismatch, thereby greatly improving the quality of synthesized speech. In addition, this approach can also enhance the system's adaptability to different speaker styles and speech rate variations, making the final generated speech not only sound more natural and fluent but also have higher stability and consistency.

[0162] In some embodiments, in order to further improve the performance of the vocoder, a step-by-step training strategy can be adopted. That is, firstly, the semantic coding features and linguistic features corresponding to the third training text are used to perform the above steps S901-S903 to perform the first stage of training and obtain the intermediate vocoder; then, the semantic coding features and linguistic features corresponding to the fourth training text are used to perform the second stage of training on the intermediate acoustic model to obtain the vocoder.

[0163] The training text for the first stage can be obtained using text data corresponding to high-quality speech data. Compared to the training text for the first stage, the training text for the second stage can increase the proportion of speaker data with the target timbre, thus enabling fine-tuning training with the target speaker.

[0164] After obtaining a trained vocoder, the acoustic features output by the acoustic model based on the concatenated semantic features can be input into the vocoder to obtain the corresponding synthesized speech.

[0165] In passing Figure 6 After training the semantic coding model in the corresponding embodiment, the speech rate of the synthesized speech output by the semantic coding model can be kept consistent with the speech rate of the speaker corresponding to the text feature block, that is, the two are kept within a small error range.

[0166] In some embodiments, in order to further reduce the delay of the preceding module and ensure the real-time performance of speech synthesis, after step S204, the method of this embodiment may further include: obtaining the delay time corresponding to the synthesized speech; if the delay time exceeds a preset delay threshold, adjusting the speech rate of the synthesized speech to obtain the speech rate-adjusted synthesized speech, wherein the time difference between the speech rate-adjusted synthesized speech and the speech time of the target speaker is within a preset time range.

[0167] In simultaneous speech interpretation scenarios, to achieve a real-time "speaking and translating" effect, the system must ensure that the playback time of the synthesized speech is synchronized with the original speaking time. That is, the time difference between the two (called "delay" or "synchronization deviation") must be controlled within a preset acceptable range (e.g., 0.5 to 1.5 seconds). If it exceeds this range, the speech rate of the synthesized speech needs to be dynamically adjusted.

[0168] For example, if the original speaker begins speaking at 10:00:02, and the system only completes speech recognition, translation, and speech synthesis at 10:00:04, then prepares to output the synthesized speech. At this point, the synthesized speech output is 2.0 seconds later than the speaking time, exceeding the preset allowable delay range of 0.5 to 1.5 seconds. If played at normal speaking speed, the listener will hear a significant delay in the translation, disrupting the real-time nature of the simultaneous interpretation and potentially interfering with the speaker's subsequent remarks. To address this, the synthesized speech speed is increased (e.g., by 1.3 times or 15% compression), allowing it to finish playing in a shorter time. This reduces the perceived delay to approximately 1.2 seconds, falling back within the preset time range and achieving "catch-up" synchronization.

[0169] When a speaker speaks too slowly, the translation and speech synthesis stages may experience excessively long waiting times, leading to output interruptions and causing the receiver to receive fragmented synthesized speech. To improve this issue and enhance the fluency of the speech output, when a speaker's slow speech rate is detected as causing disjointed synthesized output, the speech rate of the synthesized speech can be appropriately reduced to lengthen the speech output rhythm and ensure the continuity and fluency of the synthesized speech.

[0170] This embodiment employs a speed adjustment mechanism to adjust the speed of the synthesized speech after it is generated, so as to keep the overall latency within a small range and thus ensure the real-time effect of simultaneous speech interpretation.

[0171] In some embodiments, to further ensure that the speech rate of the synthesized speech is consistent with that of the original speaker, the speech rate of a real speaker can be introduced as an additional supervisory signal during the training of the acoustic model. Specifically, the speech rate of a real speaker is used as one of the training labels, forming a multi-task learning objective together with the acoustic features. In this way, the acoustic model is guided to explicitly model speech rate information during the generation of acoustic features from semantically encoded features, thereby matching the predicted acoustic features with the speaker's speech rate corresponding to the training text block in terms of temporal dynamics. This improves the naturalness and personalization of the synthesized speech in terms of rhythm, pauses, and pronunciation rate, achieving a more realistic speech reproduction effect.

[0172] Figure 11 This is a schematic diagram illustrating speech rate control training of an acoustic model, provided as an embodiment of this application. Figure 11 As shown, the acoustic model still includes encoder 801 and decoder 802; among them, encoder 801 is used to encode phoneme coding features based on phoneme sequence and sub-language sequence.

[0173] Encoder 801 is used to encode phoneme coding features based on phoneme sequences and sub-language sequences.

[0174] Decoder 802 is used to perform joint decoding based on phoneme coding features and fusion features to obtain acoustic features. These acoustic features can be Mel-spectrogram features, or multimodal acoustic parameters such as linear spectrum, F0 fundamental frequency, and energy, which are used by the subsequent vocoder to synthesize speech waveforms.

[0175] Among them, the speaker timbre features and semantic coding features after the speaker timbre information (Speaker id) has been embedded are concatenated to obtain the concatenated features (cat[spk_emb, vq_emb[1:]]); and the concatenated features are then weighted and summed with the speech speed features after the real speech speed information (Speed ​​id) has been embedded to obtain the fused features.

[0176] In this embodiment, the training process of the acoustic model includes the following steps d1-d2: Step d1: Using the acoustic model to be trained, based on the semantic encoding features and linguistic features corresponding to the training text, obtain the predicted acoustic features and their corresponding predicted speech rate information.

[0177] Continue reading Figure 11 Step d1 includes the following steps d11-d13: Step d11: By embedding the speaker timbre information corresponding to the training text, speaker timbre features are obtained.

[0178] Step d12: Fuse the speaker's timbre features and the semantic coding features corresponding to the training text to obtain the training fused features.

[0179] Step d13: Encode the training text using encoder 801 based on the phoneme sequence and sub-language sequence to obtain the phoneme encoding features corresponding to the training text.

[0180] Step d14: The decoder 802 performs joint decoding based on the phoneme encoding features and training fusion features corresponding to the training text to obtain the predicted acoustic features with predicted speech rate information.

[0181] Step d2: Based on the difference between the predicted acoustic features with predicted speech rate information and the acoustic features with real speech rate information, determine the speech rate prediction loss, and converge the acoustic model to be trained based on the speech rate prediction loss to obtain the trained acoustic model.

[0182] Based on the speech synthesis method provided in the above embodiments, this embodiment can also provide a simultaneous interpretation processing method, including: acquiring the speech to be processed of a target speaker, wherein the language of the speech to be processed is the source language; performing speech recognition on the speech to be processed to obtain a speech recognition result; translating the speech recognition result into text in the target language to obtain the text feature segment to be speech synthesized; performing speech synthesis processing on the text feature segment to be speech synthesized using the speech synthesis method provided in the above embodiments to obtain a speech synthesis result; and outputting the speech synthesis result.

[0183] In summary, this embodiment introduces historical semantic coding features as contextual memory during the predictive inference process of semantic coding features. This effectively models the temporal dependencies in speech generation, ensuring a smooth transition of intonation, rhythm, and timbre between frames, significantly improving the prosodic naturalness and auditory coherence of the synthesized speech. Furthermore, processing is done on a block-by-block basis, rather than sequentially inputting word by word or frame by frame. This allows for the early capture of local contextual information at the semantic level, reducing sensitivity to the output delay of preceding modules and effectively mitigating the latency accumulation problem caused by inter-module cascading. Further, combined with a subsequent dynamic speech rate adjustment mechanism, the pronunciation rhythm can be flexibly adjusted according to the actual synthesis progress, ensuring speech naturalness while optimizing overall response latency and enhancing the system's real-time response capability. Regarding model training, the semantic coding model, acoustic model, and vocoder all employ a phased, progressive training strategy. This not only reduces training complexity and improves convergence stability but also enhances the compatibility between modules and the overall robustness of the system, ultimately constructing a high-quality speech synthesis system that is reliable, easy to deploy, and has controllable risks.

[0184] Exemplary device Corresponding to the above-described speech synthesis method, this application also provides a speech synthesis device. Figure 12 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of this application. Figure 12 As shown, the speech synthesis apparatus provided in this application embodiment includes: an acquisition unit 1201, a determination unit 1202, and a splicing unit 1203; wherein, the acquisition unit 1201 is used to acquire a text feature block to be synthesized, the text feature block including multiple text feature frames; the determination unit 1202 is used to determine the semantic coding feature corresponding to the current text feature frame for each of the multiple text feature frames, based on the text feature block corresponding to the current text feature frame and the spliced ​​semantic coding feature corresponding to the historical text feature frames before the current text feature frame; the splicing unit 1203 is used to splice the semantic coding feature corresponding to the current text feature frame and the spliced ​​semantic coding feature corresponding to the historical text feature frames to obtain the spliced ​​semantic coding feature corresponding to the current text feature frame; the determination unit 1202 is also used to perform speech synthesis based on the spliced ​​semantic coding feature corresponding to the current text feature frame when the preset output conditions include speech synthesis based on the text feature block, and the current text feature frame is the last text feature frame among the multiple text feature frames.

[0185] In some embodiments, the semantic coding features corresponding to the current text feature frame and the concatenated semantic coding features are obtained through a semantic coding model; the semantic coding model is trained using the following steps: obtaining training text feature blocks and their corresponding training labels, wherein the training labels include real semantic coding features, and the training text feature blocks include multiple training text feature frames; for each training text feature frame, the semantic coding model to be trained predicts the current semantic coding features based on the predicted concatenated semantic coding features corresponding to the training text feature blocks and historical training text feature frames preceding the current training text feature frame, thereby determining the current training text feature frame. The predicted semantic coding features corresponding to the feature frame are obtained by concatenating the predicted semantic coding features corresponding to the current training text feature frame and the predicted concatenated semantic coding features corresponding to the historical training text feature frames. If the preset output conditions include speech synthesis based on the text feature blocks, and the current training text feature frame is the last training text feature frame among the multiple training text feature frames, then the semantic coding model to be trained is converged based on the difference between the predicted concatenated semantic coding features corresponding to the current training text feature frame and the real semantic coding features, thereby obtaining the trained semantic coding model.

[0186] In some embodiments, obtaining training text feature blocks includes: obtaining training text segments; extracting linguistic features from the training text segments to obtain training text feature segments; segmenting the training text feature segments according to preset segmentation rules to obtain various training text feature blocks; and determining each training text feature block in the various training text feature blocks as the training text feature block used for training; wherein, the linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences; and the preset segmentation rules include segmentation based on a preset number of syllables, a preset number of characters, a preset number of words, or a preset number of punctuation marks.

[0187] In some embodiments, obtaining the training text fragment includes: obtaining original training text, which includes multiple training statements; when the number of first-type training statements in the multiple training statements is greater than a preset number and the number of second-type training statements is less than a preset number, concatenating the first-type training statements so that the sentence length of the concatenated training statements is greater than or equal to a preset sentence length, thereby obtaining the training text fragment corresponding to the original training text; wherein, the first-type sentences are sentences with a sentence length less than a preset sentence length, and the second-type sentences are sentences with a sentence length greater than or equal to a preset sentence length.

[0188] In some embodiments, the training labels further include the actual speaker's speech rate; the training steps of the semantic coding model further include: predicting the speaker's speech rate corresponding to the training text feature block through the semantic coding model to be trained, to obtain the predicted speaker's speech rate; wherein, based on the difference between the predicted concatenated semantic coding features and the actual semantic coding features, the semantic coding model to be trained is converged to obtain the trained semantic coding model, including: determining the semantic coding feature prediction loss based on the difference between the predicted concatenated semantic coding features and the actual semantic coding features; determining the speaker's speech rate prediction loss based on the difference between the predicted speaker's speech rate and the actual speaker's speech rate; determining the comprehensive loss based on the sum of the semantic coding feature prediction loss and the speaker's speech rate prediction loss, and converging the semantic coding model to be trained based on the comprehensive loss to obtain the trained semantic coding model.

[0189] In some embodiments, the device further includes an adjustment unit 1204, configured to perform the following steps: obtaining the delay time corresponding to the synthesized speech; and adjusting the speech rate of the synthesized speech when the delay time exceeds a preset delay threshold, to obtain synthesized speech with adjusted speech rate, wherein the time difference between the synthesized speech with adjusted speech rate and the speech time of the target speaker is within a preset time range.

[0190] In some embodiments, the determining unit 1202 performs speech synthesis based on the spliced ​​semantic coding features, including: determining the acoustic features corresponding to the spliced ​​semantic coding features through an acoustic model; and determining the synthesized speech corresponding to the semantic coding features based on the acoustic features; wherein the acoustic model is trained using the semantic coding features and linguistic features corresponding to the training text as training samples and the real acoustic features as training labels; the linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences.

[0191] In some embodiments, the determining unit 1202 determines the synthesized speech corresponding to the semantic coding features based on the acoustic features, including: determining the synthesized speech corresponding to the semantic coding features based on the acoustic features using a vocoder; the vocoder is obtained through joint training with an acoustic model, the joint training including: predicting acoustic features based on the semantic coding features and linguistic features corresponding to the training text using the acoustic model to obtain predicted acoustic features; predicting synthesized speech based on the predicted acoustic features using the vocoder to obtain predicted synthesized speech; determining a speech synthesis prediction loss based on the difference between the predicted synthesized speech and the real synthesized speech, and converging the vocoder based on the speech synthesis prediction loss to obtain the trained vocoder.

[0192] In some embodiments, the acquisition unit 1201 acquires text feature blocks to be used for speech synthesis, including: acquiring a target text segment to be used for speech synthesis; extracting linguistic features from the target text segment to obtain text feature segments; the linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences; segmenting the text feature segments according to a preset segmentation rule to obtain various text feature blocks; the preset segmentation rule includes segmentation based on a preset number of syllables, a preset number of characters, a preset number of words, or a preset number of punctuation marks; and determining each text feature block in each text feature block as a text feature block to be used for speech synthesis; the determination unit 1202 is further configured to, when the current text feature frame is the last text feature frame among the plurality of text feature frames, determine the concatenated semantic coding feature corresponding to the current text feature frame as the semantic coding feature block corresponding to the text feature block; and the preset output condition includes performing speech synthesis based on the text feature segments, and when the current text feature block is the last text feature block among the plurality of text feature blocks, performing speech synthesis based on the semantic coding feature blocks corresponding to each text feature block.

[0193] The speech synthesis device provided in this embodiment belongs to the same concept as the speech synthesis method provided in the above embodiments of this application. It can execute the speech synthesis method provided in any of the above embodiments of this application and has the corresponding functional modules and beneficial effects for executing the speech synthesis method. Technical details not described in detail in this embodiment can be found in the specific processing content of the speech synthesis method provided in the above embodiments of this application, and will not be repeated here.

[0194] Corresponding to the simultaneous interpretation processing method described above, this application also provides a simultaneous interpretation processing apparatus. Figure 13 This is a schematic diagram of the structure of a simultaneous interpretation processing device provided in an embodiment of this application. Figure 13 As shown, the simultaneous interpretation processing device provided in this application embodiment includes: an acquisition unit 1301, a speech recognition unit 1302, a translation unit 1303, and a speech synthesis unit 1304; The system includes: an acquisition unit 1301 for acquiring the speech to be processed from the target speaker, wherein the language of the speech to be processed is the source language; a speech recognition unit 1302 for performing speech recognition on the speech to be processed to obtain a speech recognition result; a translation unit 1303 for translating the speech recognition result into text in the target language and determining the text feature blocks to be synthesized based on the text in the target language; and a speech synthesis unit 1304 for performing speech synthesis processing on the text feature blocks to be synthesized using the speech synthesis method described in the above embodiments to obtain a speech synthesis result and output the speech synthesis result.

[0195] It should be understood that the units in the above device can be implemented by a processor calling software. For example, the device includes a processor connected to a memory containing instructions. The processor calls the instructions stored in the memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. By designing the hardware circuits, some or all of the unit functions can be implemented. The hardware circuits can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all of the above units are implemented by designing the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files to implement the functions of some or all of the above units. All units in the above device can be implemented entirely by a processor calling software, entirely by hardware circuits, or partially by a processor calling software with the remaining parts implemented by hardware circuits.

[0196] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0197] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0198] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together and implemented in the form of a System-on-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and artificial intelligence processor, CPU and GPU, etc.

[0199] Exemplary electronic devices This application provides an electronic device, see [link to relevant documentation] Figure 14 As shown, the electronic device includes: Memory 200 and processor 210; The memory 200 is connected to the processor 210 and is used to store programs; The processor 210 is configured to implement the speech synthesis method disclosed in any of the above embodiments by running the program stored in the memory 200.

[0200] Specifically, the aforementioned electronic device may also include: a bus, a communication interface 220, an input device 230, and an output device 240.

[0201] The processor 210, memory 200, communication interface 220, input device 230, and output device 240 are interconnected via a bus. Among them: A bus can include a pathway for transmitting information between various components of a computer system.

[0202] Processor 210 can be a general-purpose processor, such as a general-purpose central processing unit (CPU), a microprocessor, etc., or an application-specific integrated circuit (ASIC), or one or more integrated circuits used to control the execution of the program of the present invention. It can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), an off-the-shelf programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0203] Processor 210 may include a main processor, as well as a baseband chip, modem, etc.

[0204] The memory 200 stores a program that executes the technical solution of this invention, and may also store an operating system and other key business functions. Specifically, the program may include program code, which includes computer operation instructions. More specifically, the memory 200 may include read-only memory (ROM), other types of static storage devices capable of storing static information and instructions, random access memory (RAM), other types of dynamic storage devices capable of storing information and instructions, disk storage, flash memory, etc.

[0205] Input device 230 may include a device for receiving user input data and information, such as a keyboard, mouse, camera, scanner, light pen, voice input device, touch screen, pedometer, or gravity sensor.

[0206] Output device 240 may include devices that allow information to be output to a user, such as a display screen, printer, speaker, etc.

[0207] The communication interface 220 may include a device that uses any transceiver to communicate with other devices or communication networks, such as Ethernet, Radio Access Network (RAN), Wireless Local Area Network (WLAN), etc.

[0208] The processor 210 executes the program stored in the memory 200 and calls other devices, and can be used to implement the various steps of any of the speech synthesis methods provided in the above embodiments of this application.

[0209] This application also proposes a chip, which includes a processor and a data interface. The processor reads and runs a program stored in a memory through the data interface to execute the speech synthesis method described in any of the above embodiments. For details of the processing and its beneficial effects, please refer to the embodiments of the above speech synthesis method.

[0210] Exemplary computer program products and storage media In addition to the methods and devices described above, embodiments of this application may also be computer program products, which include computer program instructions that, when executed by a processor, cause the processor to perform the steps in the speech synthesis methods according to various embodiments of this application as described in any of the above embodiments of this specification.

[0211] The computer program product can be written in any combination of one or more programming languages ​​to perform the operations of the embodiments of this application. The programming languages ​​include object-oriented programming languages ​​such as Java and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computing device, partially on the user's computing device, as a standalone software package, partially on the user's computing device and partially on a remote computing device, or entirely on a remote computing device or server.

[0212] Furthermore, embodiments of this application may also be storage media storing a computer program, which is executed by a processor to perform the steps of the speech synthesis method according to various embodiments of this application as described in any of the above embodiments of this specification.

[0213] For the foregoing method embodiments, in order to simplify the description, they are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0214] It should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For apparatus embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0215] The steps in the methods of the various embodiments of this application can be adjusted, merged, or deleted in order according to actual needs, and the technical features described in each embodiment can be replaced or combined.

[0216] The modules and sub-modules in the various embodiments of the present application's devices and terminals can be merged, divided, and deleted according to actual needs.

[0217] It should be understood that the disclosed terminals, devices, and methods can be implemented in other ways, given the several embodiments provided in this application. For example, the terminal embodiments described above are merely illustrative. For instance, the division of modules or sub-modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple sub-modules or modules may be combined or integrated into another module, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or modules, and may be electrical, mechanical, or other forms.

[0218] The modules or submodules described as separate components may or may not be physically separate. The components that constitute a module or submodule may or may not be physical modules or submodules; that is, they may be located in one place or distributed across multiple network modules or submodules. Some or all of the modules or submodules can be selected to achieve the purpose of this embodiment's solution, depending on actual needs.

[0219] Furthermore, the functional modules or sub-modules in the various embodiments of this application can be integrated into one processing module, or each module or sub-module can exist physically separately, or two or more modules or sub-modules can be integrated into one module. The integrated modules or sub-modules described above can be implemented in hardware or in the form of software functional modules or sub-modules.

[0220] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0221] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software unit executed by a processor, or a combination of both. The software unit can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0222] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0223] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A speech synthesis method, characterized in that, include: Obtain the text feature block to be used for speech synthesis, wherein the text feature block includes multiple text feature frames; For each of the multiple text feature frames, the semantic coding features corresponding to the current text feature frame are determined based on the text feature block corresponding to the current text feature frame and the concatenated semantic coding features corresponding to the historical text feature frames before the current text feature frame. The semantic coding features corresponding to the current text feature frame and the concatenated semantic coding features corresponding to the historical text feature frames are concatenated to obtain the concatenated semantic coding features corresponding to the current text feature frame. If the preset output conditions include speech synthesis based on the text feature blocks, and the current text feature frame is the last text feature frame among the multiple text feature frames, then speech synthesis is performed based on the concatenated semantic coding features corresponding to the current text feature frame.

2. The method according to claim 1, characterized in that, The semantic coding features corresponding to the current text feature frame and the concatenated semantic coding features are obtained through a semantic coding model; the semantic coding model is trained using the following steps: Obtain training text feature blocks and their corresponding training labels, wherein the training labels include real semantic encoding features and the training text feature blocks include multiple training text feature frames; For each training text feature frame in each training text feature frame, the semantic coding model to be trained predicts the current semantic coding feature based on the predicted concatenated semantic coding features corresponding to the training text feature blocks and the historical training text feature frames before the current training text feature frame, and determines the predicted semantic coding feature corresponding to the current training text feature frame. The predicted semantic coding features corresponding to the current training text feature frame and the predicted concatenated semantic coding features corresponding to the historical training text feature frames are concatenated to obtain the predicted concatenated semantic coding features corresponding to the current training text feature frame. If the preset output conditions include speech synthesis based on the text feature blocks, and the current training text feature frame is the last training text feature frame among the multiple training text feature frames, then the semantic coding model to be trained is converged based on the difference between the predicted concatenated semantic coding features corresponding to the current training text feature frame and the real semantic coding features, thereby obtaining the trained semantic coding model.

3. The method according to claim 2, characterized in that, The process of obtaining training text feature blocks includes: Obtain training text fragments; Linguistic features are extracted from the training text segments to obtain training text feature segments; The training text feature segments are segmented according to preset segmentation rules to obtain individual training text feature blocks; Each training text feature block in the respective training text feature blocks is determined as the training text feature block used for training; The linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences; The preset segmentation rules include segmentation based on a preset number of syllables, a preset number of characters, a preset number of words, or a preset number of punctuation marks.

4. The method according to claim 3, characterized in that, The acquisition of training text fragments includes: Obtain the original training text, which includes multiple training statements; If the number of first-type training statements in the plurality of training statements is greater than a preset number and the number of second-type training statements is less than a preset number, the first-type training statements are spliced ​​together so that the sentence length of the spliced ​​training statements is greater than or equal to the preset sentence length, thereby obtaining the training text fragment corresponding to the original training text. The first type of sentence is a sentence whose length is less than the preset sentence length, and the second type of sentence is a sentence whose length is greater than or equal to the preset sentence length.

5. The method according to claim 2, characterized in that, The training labels also include the actual speaker's speech rate; the training steps of the semantic encoding model also include: The speaker's speech rate is predicted by predicting the speaker's speech rate corresponding to the feature block of the training text through the semantic coding model to be trained. Specifically, based on the difference between the predicted concatenated semantic coding features and the true semantic coding features, the semantic coding model to be trained is converged to obtain the trained semantic coding model, including: Based on the difference between the predicted concatenated semantic coding features and the real semantic coding features, the semantic coding feature prediction loss is determined; Based on the difference between the predicted speaker rate and the actual speaker rate, the speaker rate prediction loss is determined; The comprehensive loss is determined by summing the semantic coding feature prediction loss and the speaker speech rate prediction loss, and the semantic coding model to be trained is converged based on the comprehensive loss to obtain the trained semantic coding model.

6. The method according to claim 1, characterized in that, After performing speech synthesis based on the concatenated semantic coding features, the method further includes: Obtain the delay time corresponding to the synthesized speech; If the delay time exceeds a preset delay threshold, the speech rate of the synthesized speech is adjusted to obtain synthesized speech with adjusted speech rate. The time difference between the synthesized speech with adjusted speech rate and the speech time of the target speaker is within a preset time range.

7. The method according to claim 1, characterized in that, The step of speech synthesis based on the concatenated semantic coding features includes: The acoustic features corresponding to the spliced ​​semantic coding features are determined by an acoustic model; Based on the acoustic features, the synthesized speech corresponding to the semantic coding features is determined; The acoustic model is trained using semantic encoding features and linguistic features corresponding to the training text as training samples and real acoustic features as training labels; the linguistic features include at least one of phoneme information, speaker timbre information and paralinguistic sequences.

8. The method according to claim 7, characterized in that, The step of determining the synthesized speech corresponding to the semantic coding features based on the acoustic features includes: The synthesized speech corresponding to the semantic coding features is determined by a vocoder based on the acoustic features; The vocoder is obtained through joint training with an acoustic model, and the joint training includes: The acoustic model predicts acoustic features based on the semantic encoding features and linguistic features corresponding to the training text, thereby obtaining predicted acoustic features. The vocoder predicts the synthesized speech based on the predicted acoustic features to obtain the predicted synthesized speech. Based on the difference between the predicted synthesized speech and the actual synthesized speech, the speech synthesis prediction loss is determined, and the vocoder is converged based on the speech synthesis prediction loss to obtain the trained vocoder.

9. The method according to claim 1, characterized in that, The process of obtaining the text feature blocks to be used for speech synthesis includes: Obtain the target text segment to be used for speech synthesis; Linguistic features are extracted from the target text segment to obtain a text feature segment; the linguistic features include at least one of phoneme information, speaker timbre information, and paralinguistic sequences. The text feature segments are segmented according to preset segmentation rules to obtain various text feature blocks; the preset segmentation rules include segmentation based on preset number of syllables, preset number of characters, preset number of words, or preset number of punctuation marks; Each text feature block in the aforementioned text feature blocks is determined as the text feature block to be used for speech synthesis; The method further includes: if the current text feature frame is the last text feature frame among the plurality of text feature frames, then determining the concatenated semantic coding feature corresponding to the current text feature frame as the semantic coding feature block corresponding to the text feature block; If the preset output conditions include speech synthesis based on the text feature segments, and the current text feature block is the last text feature block among the plurality of text feature blocks, then speech synthesis is performed based on the semantic coding feature blocks corresponding to each of the text feature blocks.

10. A method for processing simultaneous interpretation, characterized in that, include: Acquire the speech to be processed from the target speaker, wherein the language of the speech to be processed is the source language; The speech to be processed is subjected to speech recognition to obtain the speech recognition result; The speech recognition result is translated into text in the target language, and the text feature blocks to be synthesized are determined based on the text in the target language. The text feature block is processed by the speech synthesis method as described in any one of claims 1-9 to obtain a speech synthesis result; Output the speech synthesis result.

11. An electronic device, characterized in that, Including memory and processor; The memory is connected to the processor and is used to store programs; The processor is configured to implement the method as described in any one of claims 1 to 10 by running a program in the memory.

12. A computer program product, characterized in that, It includes computer program instructions that, when executed by a processor, cause the processor to perform the method as described in any one of claims 1 to 10.

Images