A speech synthesis method, device, computer equipment and storage medium

Abstract

Embodiments of the present disclosure relate to the technical field of speech processing, and specifically relate to a speech synthesis method and device, computer equipment and a storage medium. The main steps of the foregoing method include: obtaining text data to be synthesized, performing language recognition on the text data, and determining at least one target language category to which the text data belongs. Based on the at least one target language category, the text data is subjected to text normalization processing to obtain normalized text. The unified phoneme sequence corresponding to the normalized text is input into a pre-trained shared acoustic model to obtain target acoustic features; and based on the target acoustic features, synthesized speech is obtained. By using a unified phoneme sequence defined based on cross-language pronunciation features as an intermediate identifier, a single shared acoustic model is used for processing, which significantly reduces the overall parameter quantity, storage space requirement and computing resource consumption of the system, thereby directly reducing the deployment and operation costs.

Description

Technical Field

[0001] This disclosure relates to the field of speech processing technology, and in particular to a speech synthesis method, apparatus, computer device, and storage medium. Background Technology

[0002] With the development of artificial intelligence and speech processing technologies, speech synthesis technology based on deep neural networks has emerged. This technology can convert text into natural and fluent speech and has been widely applied in various scenarios such as intelligent voice assistants, in-vehicle navigation, and accessible reading. Existing mainstream speech synthesis models can generate high-quality speech within a single language or a limited range of languages.

[0003] However, with the acceleration of globalization and the increasing demand for cross-language communication, existing speech synthesis technologies face significant challenges in supporting multiple languages ​​simultaneously and maintaining high efficiency. For example, training a complete speech synthesis model separately for each language would result in a massive overall model size, a surge in storage and computing resource requirements, and difficulty in deployment in resource-constrained environments such as mobile devices and embedded devices. On the other hand, using a single model with completely shared parameters to process all languages ​​would make it difficult to accurately capture the specific differences between languages, leading to a decrease in the naturalness and accuracy of the synthesized speech. Summary of the Invention

[0004] Therefore, it is necessary to provide a speech synthesis method, apparatus, computer device, and storage medium to address at least one of the aforementioned technical problems.

[0005] In a first aspect, embodiments of this disclosure provide a speech synthesis method, the method comprising: The text data to be synthesized is acquired, and language recognition is performed on the text data to determine at least one target language category to which the text data belongs.

[0006] Based on at least one target language category, the text data is processed to obtain normalized text.

[0007] The unified phoneme sequence corresponding to the normalized text is input into a pre-trained shared acoustic model to obtain the target acoustic features; where each unified phoneme in the unified phoneme sequence is a speech unit defined by a set of pronunciation features shared across languages.

[0008] Synthetic speech is obtained based on the acoustic features of the target.

[0009] In some embodiments, the speech synthesis method further includes: Convert normalized text into a sequence of original phonemes corresponding to at least one target language category; According to the preset mapping relationship, each original phoneme in the original phoneme sequence is mapped to a unified phoneme in the preset unified phoneme set to obtain a unified phoneme sequence; wherein, the mapping relationship is established based on at least one of the following: place of articulation, manner of articulation, voicing attribute, and tone attribute.

[0010] In some embodiments, the speech synthesis method further includes: The training samples are input into the pre-set teacher model to obtain reference acoustic features; the teacher model is an acoustic model trained based on a unified phoneme set, and the number of parameters in the teacher model is greater than that in the pre-trained shared acoustic model. Using training samples and reference acoustic features as supervision signals, the initial shared acoustic model is trained to obtain a pre-trained shared acoustic model.

[0011] In some embodiments, the pre-trained shared acoustic model includes a shared coding network and at least one language adaptation module; The process of inputting the unified phoneme sequence corresponding to the normalized text into a pre-trained shared acoustic model to obtain the target acoustic features also includes the following steps: The unified phoneme sequence is propagated layer by layer between the network layers of the shared coding network. During the propagation process, a language adaptation module corresponding to at least one target language category is invoked to perform feature transformation on the intermediate features output by the current network layer, and finally output the target acoustic features.

[0012] In some embodiments, the speech synthesis method further includes: Predict the acoustic variation information corresponding to the target acoustic features; the acoustic variation information includes at least the duration information, fundamental frequency information and energy information of each unified phoneme in the unified phoneme sequence; The target acoustic features are processed based on acoustic change information to adjust the prosodic features of the synthesized speech; the prosodic features include at least speech rate and intonation.

[0013] In some embodiments, synthesizing and outputting synthesized speech based on target acoustic features further includes the following steps: The target acoustic features are segmented into blocks to generate corresponding speech waveform blocks; Adjacent speech waveform blocks are spliced ​​together to generate a continuous speech waveform, which is then used to output synthesized speech.

[0014] In some embodiments, the speech synthesis method further includes: Obtain the reference speech of the target object and extract the timbre feature vector from the reference speech; The target acoustic features are adjusted based on the timbre feature vector to determine the timbre of the synthesized speech.

[0015] In a second aspect, embodiments of this disclosure provide a speech synthesis apparatus, the apparatus comprising: The recognition module is used to acquire the text data to be synthesized, perform language recognition on the text data, and determine the target language category to which the text data belongs; The normalization module is used to perform text normalization processing on text data based on at least one target language category to obtain normalized text; The feature acquisition module is used to input the unified phoneme sequence corresponding to the normalized text into a pre-trained shared acoustic model to obtain the target acoustic features; wherein, each unified phoneme in the unified phoneme sequence is a speech unit defined by a set of pronunciation features shared across languages; The synthesis module is used to obtain synthesized speech based on the target acoustic features.

[0016] In a third aspect, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the speech synthesis method provided in any embodiment of the first aspect of this disclosure.

[0017] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech synthesis method provided in any embodiment of the first aspect of the present disclosure.

[0018] The aforementioned speech synthesis method, apparatus, computer equipment, and storage medium employ a unified phoneme sequence defined based on cross-language pronunciation features as an intermediate identifier and utilize a single shared acoustic model for processing. Since only one shared acoustic model needs to be maintained and run, rather than multiple language-specific models, this significantly reduces the overall system parameter count, storage space requirements, and computational resource consumption, thereby directly reducing deployment and operating costs and improving the computational efficiency of the speech synthesis process. The pre-processing language identification and targeted text normalization steps ensure the accuracy and compliance of different language texts in the conversion to unified phonemes, guaranteeing the speech quality and stability of multilingual synthesis from the front end of the process. Attached Figure Description

[0019] Figure 1 This is a diagram illustrating the application environment of the speech synthesis method in some embodiments; Figure 2 This is a flowchart illustrating the speech synthesis method in some embodiments; Figure 3 This is a flowchart illustrating the steps involving the original phoneme sequence in some embodiments; Figure 4 This is a flowchart illustrating the steps involved in the teacher model in some embodiments; Figure 5 This is a flowchart illustrating the steps involved in predicting acoustic change information in some embodiments; Figure 6 This is a flowchart illustrating the segmentation and splicing steps in some embodiments; Figure 7 This is a flowchart illustrating the timbre steps in some embodiments; Figure 8 Here are some block diagrams of the speech synthesis apparatus in the embodiments; Figure 9 This is a diagram showing the internal structure of a computer device in some embodiments. Detailed Implementation

[0020] To make the technical solutions and advantages of this application clearer, the embodiments and related technical content of this application will be further described in detail below with reference to the accompanying drawings and text description. It should be understood that the embodiments described below are only used to explain the technical solutions of the embodiments of this application and are not intended to limit more possible implementations of this application.

[0021] It should be noted that, in the description of this application, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0022] For ease of understanding, Figure 1 An application environment is illustrated, in which server 110 executes the steps of a speech synthesis method. During execution, server 110 can communicate with terminal 120 via a network to obtain speech synthesis-related data sent by terminal 120. Server 110 can be implemented as a standalone server or a server cluster consisting of multiple servers. Terminal 120 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, or portable wearable devices.

[0023] Servers can be implemented using at least one of the following hardware forms: programmable logic array (PLA), field-programmable gate array (FPGA), digital signal processor (DSP), application-specific integrated circuit (ASIC), general-purpose processor, or other programmable logic device.

[0024] Of course, the speech synthesis method provided in this application embodiment can also be applied to more scenarios not shown.

[0025] In a first aspect, embodiments of this application provide a speech synthesis method that can be applied to... Figure 1In the application environment shown, it can be applied to Figure 1 Taking server 110 as an example, in some embodiments, such as Figure 2 As shown, the speech synthesis method includes steps S210, S220, S230, and S240 that can be executed by the server 110. Each step is described in detail below.

[0026] Step S210: Obtain the text data to be synthesized, perform language recognition on the text data, and determine the at least one target language category to which the text data belongs.

[0027] Language recognition can refer to the language discrimination process based on character-level or word-level statistical features, context modeling capabilities, or lightweight neural network structures. Its input is raw text data, such as text strings, and its output is one or more language labels and their confidence scores.

[0028] Specifically, the text data to be synthesized can be segmented by sentences or preset windows, and BiLSTM (Bidirectional Long Short-Term Memory) or a simplified lightweight Transformer can be used to perform language recognition and classification on each character or word in the text data. For cases involving a mix of different language categories, markers can be added at language switching points without altering the original punctuation.

[0029] At least one target language category is used to guide the selection of text normalization and phoneme mapping strategies. For example, when the recognition result is a mixture of Chinese and English, the text normalization rules corresponding to Chinese and English are called respectively, and the mapping relationship between the original phonemes and unified phonemes of Chinese and English is called respectively in the unified phoneme sequence generation stage.

[0030] For example, assuming the text data is "Turn left in 200 meters", the text is first segmented into characters and encoded using a sliding window. The language probability distribution of each window is output by the BiLSTM network. Then, through context aggregation and threshold determination, it is identified that the text contains two languages: English and Chinese, with confidence levels of 0.92 and 0.87, respectively. Based on this, the target language category is determined to be English and Chinese.

[0031] Step S220: Based on at least one target language category, perform text normalization processing on the text data to obtain normalized text.

[0032] Text normalization can refer to the process of converting non-standard text forms (such as numbers, dates, abbreviations, units, etc.) into standard pronunciations that conform to speech reading habits.

[0033] Specifically, text normalization processing based on the target language category can refer to selecting the corresponding language rule engine or neural sub-model to perform normalization operations based on the identified language tags. For example, the English number 200 can be transcribed into "two hundred" according to English rules, and the Chinese unit of meter can be retained in its original form and labeled as "mǐ". This ensures that texts of different languages ​​have completed semantic alignment and pronunciation rule adaptation before entering the phoneme conversion stage, avoiding pronunciation errors or rhythmic breaks caused by format ambiguity.

[0034] For example, this application can be for the bilingual English and Chinese scenarios identified above. The system calls the English rule module to transcribe 200 into two hundred, and at the same time calls the Chinese rule module to standardize the rice into pinyin mǐ, and organizes the whole sentence into turn left in two hundred mǐ. The result is the standardized text, which not only preserves the semantic structure of the original text, but also meets the pronunciation standards of each language.

[0035] Step S230: Input the unified phoneme sequence corresponding to the normalized text into the pre-trained shared acoustic model to obtain the target acoustic features.

[0036] In this context, each unified phoneme in the unified phoneme sequence is a phonetic unit defined by a set of pronunciation features shared across languages.

[0037] A Unified Phone Set (UPS) can refer to a set of universal speech units that are cross-language based on the International Phonetic Alphabet (IPA) and integrate articulation dimensions such as place of articulation, manner of articulation, voicing, tone, stress, nasalization, retroflexion, and aspiration.

[0038] A unified phoneme is a speech unit defined by a set of pronunciation features shared across languages. This means that the same unified phoneme can correspond to multiple original phonemes in different languages. For example, UPS_id=101 covers both Chinese p and English p, but its underlying articulation feature vector remains consistent.

[0039] Step S240: Obtain synthesized speech based on target acoustic features.

[0040] Target acoustic features refer to intermediate representations of speech spectral characteristics output by a shared acoustic model, such as Mel frequency cepstral coefficients or Mel scale logarithmic spectra.

[0041] Obtaining synthesized speech based on target acoustic features can refer to feeding these acoustic features into a subsequent vocoder module, reconstructing the waveform, and outputting a playable continuous speech signal. This process does not rely on language-specific decoders; all languages ​​share the same vocoder architecture, thereby reducing system complexity and deployment costs.

[0042] By employing a unified phoneme sequence defined based on cross-language pronunciation features as an intermediate identifier and utilizing a single shared acoustic model for processing, the overall system parameter count, storage space requirements, and computational resource consumption are significantly reduced because only one shared acoustic model needs to be maintained and run, rather than multiple language-specific models. This directly lowers deployment and operating costs and improves the computational efficiency of the speech synthesis process. The pre-processing language identification and targeted text normalization steps ensure the accuracy and compliance of different language texts in the conversion to the unified phoneme sequence, guaranteeing the speech quality and stability of multilingual synthesis from the front end of the process.

[0043] In some embodiments, such as Figure 3 As shown, the speech synthesis method may further include steps S310 and S320.

[0044] Step S310: Convert the normalized text into the original phoneme sequence corresponding to at least one target language category.

[0045] Original phonemes can refer to discrete symbols defined according to the target language's own phonetic system, used to represent the pronunciation units of that language.

[0046] Specifically, for standardized text, based on the identified target language category, a language-specific text-to-phoneme module is invoked, such as a pinyin converter for Chinese or a grapheme-to-phoneme (G2P) model for English, to generate the original phoneme sequence that conforms to the language's conventions. This ensures that the text for each language is first correctly parsed into its most natural and accurate pronunciation representation.

[0047] When the normalized text involves multiple languages, the original phoneme sequence is generated in segments according to the language detection results, and language identifiers are inserted at the language switching boundaries to maintain the integrity of the phoneme context.

[0048] Step S320: According to the preset mapping relationship, map each original phoneme in the original phoneme sequence to a unified phoneme in the preset unified phoneme set to obtain a unified phoneme sequence.

[0049] The mapping relationship is established based on at least one of the following: place of articulation, manner of articulation, voicing, and tone.

[0050] To achieve cross-language processing, it is necessary to establish correspondences between phoneme systems of different languages. These mappings can be manually defined using linguistic knowledge or automatically obtained through statistical alignment of bilingual speech data. The mapping defines how to convert a language-specific phoneme symbol into a common, more abstract phoneme identifier.

[0051] The pre-defined unified phoneme set is a finite, discrete library of units, each unit corresponding to one or more combinations of pronunciation features. The mapping process essentially extracts the phonetic attributes of the original phonemes and matches them to units in the unified phoneme set that have the same or most similar attributes.

[0052] For example, the English / t / (voiceless, alveolar, plosive) and the Chinese / t / (voiceless, alveolar, plosive) might be mapped to the same Uniform Phoneme ID; while the English / p / (voiceless, bilabial, plosive, unaspirated) and the Chinese / p / ... Although the / (voiceless, bilabial, plosive, aspirated) sounds share the same place and manner of articulation, they may be mapped to two different unified phoneme IDs due to their different aspiration properties. This feature-based mapping allows unified phoneme sequences to accurately preserve the articulatory essence of the original speech.

[0053] Each UPS unit in the Unified Phoneme Set can include the following fields: UPS_id (unique unified phoneme identifier), International Phonetic Alphabet (IPA), place (place of articulation), manner (manner of articulation), voiced (voicing), aspiration (aspiration), nasal (nasalization), retroflex (retroflexion), length (length / short), tone (tone), stress (stress), lang_mask (supported language set), and alias (original language phoneme). The Unified Phoneme Set can be stored as a database or dictionary table structure.

[0054] By first converting the standardized text into the original phoneme sequence corresponding to the target language, and then using a mapping relationship established based on at least one of the articulation place, articulation mode, voicing attribute, and tone attribute to complete the structured mapping to a unified phoneme set, the distortion risk caused by cross-language phoneme direct translation is avoided, and the interpretability loss caused by relying entirely on data-driven approaches is also avoided. The unified phoneme sequence formed on this basis can accurately carry the essential differences in pronunciation between different languages, providing a language-independent intermediate representation for shared acoustic models, thereby supporting high-quality, low-resource-consumption multilingual speech synthesis.

[0055] In some embodiments, such as Figure 4 As shown, the speech synthesis method may further include steps S410 and S420.

[0056] Step S410: Input the training samples into the preset teacher model to obtain reference acoustic features.

[0057] The teacher model is an acoustic model trained on a unified set of phonemes, and the number of parameters in the teacher model is greater than that in the pre-trained shared acoustic model.

[0058] Knowledge distillation is a model compression technique that typically involves a large, complex, but high-performance teacher model and a smaller, efficient student model. The teacher model learns complex input-output mappings through extensive training on large amounts of data.

[0059] In this application, the pre-defined teacher model is itself a multilingual acoustic model. Its input is a large number of unified phoneme sequence samples corresponding to the training text in the unified phoneme set. By training on a large number of multilingual unified phoneme sequence samples, the teacher model can learn the complex mapping from unified phonemes to acoustic features.

[0060] Step S420: Using training samples and reference acoustic features as supervision signals, train the initial shared acoustic model to obtain a pre-trained shared acoustic model.

[0061] In distillation training, the student model targets not only the original training data labels, but also the outputs (soft labels or intermediate features) produced by the teacher model on the same input data, serving as a richer learning objective. This process allows the student model to mimic the behavior of the teacher model, thereby achieving near-teacher model performance with fewer parameters.

[0062] Understandably, the shared acoustic model is the student model. The training objective of the student model is to make its own output acoustic features as close as possible to the reference acoustic features output by the teacher model. This allows the student model to directly learn the high-quality sound generation patterns already refined by the teacher model, rather than learning from scratch a raw, potentially noisy, data distribution. This process does not change the teacher model parameters; it only updates the trainable parameters of the initial shared acoustic model.

[0063] In some alternative embodiments, intermediate features generated by the teacher model can also be used as supervision signals for training the student model, so that the performance of the shared acoustic model can be further close to that of the teacher model.

[0064] In addition, operations such as quantization, pruning, and low-rank decomposition can be performed on the trained student model (i.e., the shared acoustic model). Without significantly sacrificing performance, the student model can be compressed to adapt to environments with demanding algorithms, memory, and power consumption, such as mobile phones and low-end IoT devices.

[0065] By constructing a powerful teacher model and using its high-quality acoustic features as supervision signals, a clear and efficient optimization objective is provided for training a lightweight shared acoustic model (student model), enabling the small model to directly learn complex acoustic patterns already mastered by the large model. The knowledge distillation technique is used for training, so that the final shared acoustic model can achieve speech synthesis quality close to or even comparable to that of the large teacher model while maintaining a small number of parameters and low computational complexity, thus balancing model efficiency and speech synthesis quality.

[0066] In some embodiments, the pre-trained shared acoustic model includes a shared coding network and at least one language adaptation module. Based on this, step S230 may further include the following steps: propagating the unified phoneme sequence layer by layer between the network layers of the shared coding network, and calling the language adaptation module corresponding to at least one target language category during the propagation process to perform feature transformation on the intermediate features output by the current network layer, and finally outputting the target acoustic features.

[0067] A shared coding network can refer to a backbone neural network structure used for feature extraction from a uniform phoneme sequence, whose parameters are completely shared across all language categories. A shared coding network can be a lightweight Transformer or Conformer architecture, containing L stacked coding layers, each with a consistent output dimension, and without introducing language-specific parameters.

[0068] A language adaptation module can be a lightweight, learnable submodule deployed within a shared coding network and bound to a specific target language category. It applies language-specific adjustments to intermediate layer features without altering the main parameters of the shared coding network. The language adaptation module can be any of the following: a bottleneck structure Adapter, a low-rank incremental LoRA, or a sparse gated MoE.

[0069] Calling a language adaptation module corresponding to at least one target language category means that, based on at least one determined target language category, when the unified phoneme sequence is propagated forward to a certain layer, the language adaptation module matching the target language category is activated according to the input conditions. When there are multiple target language categories, multiple language adaptation modules can be called in parallel, or they can be called by paragraph according to the language boundary label.

[0070] Feature transformation of the intermediate features output by the current network layer can refer to taking the original output feature vector of the network layer as input, performing linear projection, nonlinear activation, and residual fusion through the adaptation module, and then outputting updated intermediate features. This transformation does not change the feature dimension, but only adjusts its distribution to enhance language discriminativeness and pronunciation specificity.

[0071] The final output target acoustic features can refer to a high-dimensional continuous vector sequence output by the shared acoustic model after all L layers of encoding and language adaptation operations of the corresponding number of layers.

[0072] By decomposing the shared acoustic model into a shared coding network and a language adaptation module, model parameters can be reused. The shared coding network, which contains the majority of the model's parameters, is shared by all languages, significantly reducing the total number of parameters and achieving efficient storage and computation. The language adaptation module, as a lightweight, pluggable component, allows the model to adapt to the synthesis needs of different languages ​​by activating different adaptation modules without changing the core shared parameters. When a new language is added, only a new set of language adaptation modules with a very small number of parameters needs to be trained for that language and inserted into a fixed position in the existing shared coding network, without retraining the entire large model. This significantly reduces expansion costs and shortens the development cycle.

[0073] In some embodiments, such as Figure 5 As shown, the speech synthesis method may further include steps S510 and S520.

[0074] Step S510: Predict the acoustic change information corresponding to the target acoustic features.

[0075] The acoustic variation information includes at least the duration, fundamental frequency, and energy information of each unified phoneme in the unified phoneme sequence.

[0076] Acoustic variation information can refer to a set of auxiliary acoustic variables used to characterize the dynamic characteristics of speech prosody. In speech synthesis systems, these variables play a role in regulating the basic acoustic features at the rhythm and intonation levels.

[0077] Duration information can refer to the target frame number or duration corresponding to a uniform phoneme on the time axis, used to control the pronunciation duration of that uniform phoneme in the final speech waveform. Its function is to ensure that different uniform phonemes in the normalized text present a rhythmic distribution that conforms to language habits in the output synthesized speech, thereby affecting the overall speech rate.

[0078] Fundamental frequency information refers to periodic characteristic parameters that reflect the frequency of vocal cord vibration and are used to characterize the pitch variation trend of speech. Its role is as the core input for intonation modeling. By injecting fundamental frequency information contours into the target acoustic features, synthesized speech can have intonation patterns such as rising, falling, and flat, thereby supporting semantic expressions such as questions, statements, and emphasis.

[0079] Energy information refers to the statistical measure of the amplitude of the speech signal within a unit time window, used to characterize the stress, volume, and emotional intensity of speech. Its role is to work in conjunction with fundamental frequency information to regulate the prosodic focus of speech, enhance the perception of stressed syllables, and improve the naturalness of cross-linguistic speech in semantic emphasis and emotional expression.

[0080] For example, a dedicated prediction module can be designed, including a duration prediction submodule, a fundamental frequency prediction submodule, and an energy prediction submodule. This prediction module takes a unified phoneme sequence or target acoustic features as input and predicts duration information, fundamental frequency information, and energy information for each unified phoneme in the unified phoneme sequence in parallel.

[0081] The duration prediction submodule can be used to perform regression estimation on the target acoustic features to obtain the duration value corresponding to each unified phoneme; or, the duration prediction submodule can be used to perform conditional regression on the target acoustic features in combination with the articulation attributes (such as articulation mode and voicing attributes) of the unified phoneme to obtain the duration value corresponding to each unified phoneme.

[0082] The fundamental frequency prediction submodule can be used to perform continuous value regression on the target acoustic features to obtain the fundamental frequency trajectory corresponding to each unified phoneme; or, the fundamental frequency can be divided into discrete bins and a classification task can be performed based on the fundamental frequency prediction submodule, and then restored to a continuous fundamental frequency sequence through embedding mapping.

[0083] The energy prediction submodule can be used to perform window-level root mean square energy estimation on the target acoustic features to obtain the energy scalar corresponding to each unified phoneme; or, the target acoustic features can be hierarchically allocated according to the positional relationship of the unified phonemes in the word / phrase level.

[0084] Step S520: Process the target acoustic features based on acoustic change information to adjust the prosodic features of the synthesized speech.

[0085] Among them, prosodic features include at least speech rate and intonation.

[0086] Speech rate refers to the information density of spoken content per unit of time, which is determined by the overall duration distribution of a unified phoneme sequence. By adjusting the duration information of each unified phoneme, the overall speech rate can be increased or decreased, or effects such as linking or pausing can be achieved.

[0087] Intonation refers to the overall contour of the fundamental frequency of speech over time, which is determined by the sequential distribution of fundamental frequency information. By adjusting the start, transition, and end points of the fundamental frequency information, rising, falling, level, and compound tone patterns can be constructed, thereby conveying semantic functions such as questioning, affirmation, and surprise.

[0088] Specifically, based on the prediction results, the target acoustic features are resampled or frames are copied / deleted on the time axis to ensure that the duration of the spectrogram matches the predicted duration, thereby controlling the speech rate. Based on the predicted fundamental frequency information curve and energy information curve, these are fused into the duration-adjusted target acoustic features through feature splicing or frame-by-frame weighting. This encodes the pitch (determined by the fundamental frequency information curve) and stress / loudness (determined by the energy information curve) information into the final target acoustic features.

[0089] By predicting the duration, fundamental frequency, and energy information of each unified phoneme in the unified phoneme sequence, and performing length extension and multi-dimensional fusion processing on the target acoustic features based on this acoustic change information, the synthesized speech has the ability to control the prosodic rate and intonation. On this basis, by using the articulation attributes of the unified phonemes and language tags, language consistency constraints are imposed on the prosodic prediction process, so that the synthesized speech does not lose any single language expressiveness and naturalness in the case of multilingual speech.

[0090] In some embodiments, such as Figure 6 As shown, the speech synthesis method may further include steps S610 and S620.

[0091] Step S610: Perform block processing on the target acoustic features to generate corresponding speech waveform blocks.

[0092] Block processing can refer to dividing the target acoustic features along the time axis into multiple continuous subsequences with controllable lengths. Each subsequence constitutes a processing unit. When a processing unit is input to the vocoder, it can independently trigger a waveform generation process. The generation of multiple speech waveform blocks can be synchronous or asynchronous.

[0093] For example, the target acoustic features can be divided into blocks according to a sliding window strategy, where the window length is a fixed number of frames (e.g., 32, 64, or 128 frames), and the window step size is smaller than the window length to achieve overlap; or, the block boundaries can be dynamically divided according to speech semantic boundaries (e.g., phoneme boundaries, prosodic word boundaries, or pause prediction results) so that each block covers the complete articulation unit as much as possible.

[0094] Step S620: Concatenate adjacent speech waveform blocks to generate a continuous speech waveform, and output synthesized speech.

[0095] Adjacent speech waveform blocks refer to two speech waveform blocks that are sequentially adjacent on the time axis of the target acoustic features and are generated one after the other after block processing.

[0096] Splicing can refer to connecting the tail of one speech waveform block to the head of the next speech waveform block in chronological order, and applying a smooth transition process in the overlapping area to eliminate the temporal discontinuities introduced by the block segmentation. Smoothing transition processes include, but are not limited to, weighted averaging, crossfade-in / fade-out, phase continuity correction, or overlapping windowing reconstruction based on short-time Fourier transform.

[0097] For example, adjacent speech waveform blocks can be spliced ​​using a weighted average according to a preset overlap length (e.g., 16ms or 32ms), with the weights varying linearly with the overlap position.

[0098] This application describes a scenario in vehicle navigation where the system receives text indicating a right turn 500 meters ahead. After processing through speech recognition, text normalization, unified phoneme sequence generation, and a shared acoustic model, a target acoustic feature (80-dimensional spectral resolution, 10ms frame shift) with a duration of 1.8 seconds is obtained. This target acoustic feature is divided into four speech blocks in units of 64 frames (640ms). Each block is input to a HiFi-GAN vocoder to generate a corresponding speech waveform block with a duration of approximately 640ms. A 32ms overlap is maintained between adjacent speech waveform blocks, and they are spliced ​​using a 20ms cross-fade-in / fade-out method. The final output is a continuous synthesized speech with a total duration of approximately 1.8 seconds and an end-to-end latency controlled within 220ms, meeting the low-latency interaction requirements in a vehicle environment.

[0099] By segmenting the target acoustic features and driving the vocoder to generate speech waveform blocks, and then using an overlapping splicing mechanism to ensure temporal continuity, combined with collaborative mechanisms such as sliding window partitioning, semantically aware segmentation and phase-sensitive splicing, the inherent latency bottleneck of traditional end-to-end vocoders that must wait for the entire sequence of input before starting synthesis is avoided. Thus, without increasing the number of model parameters, the real-time response and flexibility of the speech synthesis system are significantly improved. It is particularly suitable for application scenarios with limited resources and strong timeliness requirements, such as mobile terminals, embedded devices and in-vehicle voice interaction.

[0100] In some embodiments, such as Figure 7 As shown, the speech synthesis method may further include steps S710 and S720.

[0101] Step S710: Obtain the reference speech of the target object and extract the timbre feature vector from the reference speech.

[0102] Reference speech can refer to a speech passage read by the target, which can be 10 to 60 seconds long. It can be a general sentence with neutral intonation and covering the distribution of vowels and consonants, such as "The weather is nice today" or "Hello, this is a test sentence".

[0103] A timbre feature vector can be a low-dimensional, fixed-length vector used to characterize the individual acoustic characteristics of a speaker. It can be used as a voiceprint identifier for a target object, and it is independent of language content and has cross-language consistency.

[0104] For example, frame-level acoustic feature extraction and statistical pooling of the reference speech can be performed based on a pre-trained speaker encoder to obtain a timbre feature vector; alternatively, a normalized embedding vector can be directly output after inputting the reference speech based on a pre-trained end-to-end speaker verification model; or a lightweight convolutional network can be used to perform temporal modeling of the Mel spectrum sequence and generate a timbre feature vector through attention-weighted pooling.

[0105] Step S720: Adjust the target acoustic features based on the timbre feature vector to determine the timbre of the synthesized speech.

[0106] Adjusting the target acoustic features can refer to incorporating the timbre feature vector as global contextual information into the generation process of the target acoustic features, so that it can carry the timbre attributes of the target object while maintaining the accuracy of the language pronunciation; For example, the timbre feature vector can be linearly projected and then broadcast-added with the target acoustic features in the time dimension to obtain the timbre-enhanced acoustic features; alternatively, the timbre feature vector can be used as a conditional input to control the parameter bias of the duration prediction submodule and the fundamental frequency prediction submodule, thereby indirectly affecting the prosodic structure of the target acoustic features; or the timbre feature vector can be concatenated with a unified phoneme sequence at the input layer and fed into a shared coding network, so that the timbre information participates in feature evolution from the starting point of modeling.

[0107] By using the timbre feature vector extracted from the reference speech of the target object as conditional information, the target acoustic features output by the shared acoustic model are dynamically adjusted, enabling the same multilingual speech synthesis model to output speech results with a unified speaker timbre under different language text inputs. Based on this, there is no need to build a separate timbre-customized model for each language, nor is it dependent on large-scale target language speech data. Cross-language timbre transfer can be achieved with only a small amount of reference speech, significantly improving the flexibility of personalized speech synthesis.

[0108] It should be understood that, although Figures 2 to 7 The steps in the flowchart are shown sequentially according to the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Figures 2 to 7Unless otherwise expressly stated herein, the steps illustrated and other steps involved in the embodiments are not subject to strict order restrictions and may be performed in other orders. Furthermore, at least some steps in the foregoing embodiments may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but may be performed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0109] In a second aspect, embodiments of this disclosure provide a speech synthesis apparatus, such as Figure 8 As shown, the speech synthesis device 800 includes: a recognition module 810, a normalization module 820, a feature acquisition module 830, and a synthesis module 840.

[0110] The recognition module 810 is used to acquire the text data to be synthesized, perform language recognition on the text data, and determine the target language category to which the text data belongs.

[0111] The normalization module 820 is used to perform text normalization processing on text data based on at least one target language category to obtain normalized text.

[0112] The feature acquisition module 830 is used to input the unified phoneme sequence corresponding to the normalized text into the pre-trained shared acoustic model to obtain the target acoustic features; wherein, each unified phoneme in the unified phoneme sequence is a speech unit defined by a set of pronunciation features shared across languages.

[0113] Synthesis module 840 is used to obtain synthesized speech based on target acoustic features.

[0114] In some embodiments, the speech synthesis apparatus further includes: The conversion module (not shown) is used to convert normalized text into a sequence of original phonemes corresponding to at least one target language category; The mapping module (not shown) is used to map each original phoneme in the original phoneme sequence to a unified phoneme in a preset unified phoneme set according to a preset mapping relationship, thereby obtaining a unified phoneme sequence; wherein, the mapping relationship is established based on at least one of the following: place of articulation, manner of articulation, voicing attribute, and tone attribute.

[0115] In some embodiments, the speech synthesis device 800 further includes: The first training module (not shown) is used to input training samples into a pre-set teacher model to obtain reference acoustic features; wherein, the teacher model is an acoustic model trained based on a unified phoneme set, and the number of parameters of the teacher model is greater than that of the pre-trained shared acoustic model; The second training module (not shown) is used to train the initial shared acoustic model using training samples and reference acoustic features as supervision signals to obtain a pre-trained shared acoustic model.

[0116] In some embodiments, the pre-trained shared acoustic model includes a shared coding network and at least one language adaptation module; The feature acquisition module 830 also includes: The feature propagation unit propagates the unified phoneme sequence layer by layer between the network layers of the shared coding network. During the propagation process, it calls the language adaptation module corresponding to at least one target language category to perform feature transformation on the intermediate features output by the current network layer, and finally outputs the target acoustic features.

[0117] In some embodiments, the speech synthesis device 800 further includes: The prediction module (not shown) is used to predict the acoustic change information corresponding to the target acoustic features; the acoustic change information includes at least the duration information, fundamental frequency information and energy information of each unified phoneme in the unified phoneme sequence; The prosodic processing module (not shown) is used to process the target acoustic features based on acoustic change information in order to adjust the prosodic features of the synthesized speech; the prosodic features include at least speech rate and intonation.

[0118] In some embodiments, the synthesis module 840 includes: The segmentation unit (not shown) is used to segment the target acoustic features into blocks to generate corresponding speech waveform blocks; The splicing unit (not shown) is used to splice adjacent speech waveform blocks to generate a continuous speech waveform for outputting synthesized speech.

[0119] In some embodiments, the speech synthesis device 800 further includes: The timbre extraction module (not shown) is used to obtain the reference speech of the target object and extract the timbre feature vector from the reference speech; The timbre adjustment module (not shown) is used to adjust the target acoustic features based on the timbre feature vector in order to determine the timbre of the synthesized speech.

[0120] For further specific limitations regarding the speech synthesis device, please refer to the limitations of the speech synthesis method above. The speech synthesis device can also be used to perform more steps of the speech synthesis method in the embodiments of this disclosure, which will not be repeated here. Each module in the above-described speech synthesis device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0121] In a third aspect, embodiments of this disclosure provide a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the speech synthesis method provided in any embodiment of the first aspect of this disclosure.

[0122] In some embodiments, the computer device may be a server, and its internal structure diagram may be as follows: Figure 9 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores speech synthesis-related data. The network interface communicates with external terminals via a network connection. When executed by the processor, the computer program implements the speech synthesis method in any embodiment of this document.

[0123] Those skilled in the art will understand that Figure 9 The structures shown are merely block diagrams of some structures related to the embodiments of this disclosure and do not constitute a limitation on the computer devices to which the embodiments of this disclosure are applied. Specific computer devices may include more or fewer components than those shown in the figures, or combine certain components, or have different component arrangements.

[0124] In a fourth aspect, embodiments of the present disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the speech synthesis method provided in any embodiment of the first aspect of the present disclosure.

[0125] The computer-readable storage medium may be Figure 9 The non-volatile storage medium in the computer device shown.

[0126] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The aforementioned computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments of this disclosure can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0127] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this disclosure.

[0128] The above embodiments merely illustrate several implementation methods of this disclosure, and their descriptions are relatively specific and detailed, but they should not be construed as limiting the scope of protection of this disclosure. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this disclosure, and these all fall within the scope of protection of this disclosure. Therefore, the scope of protection of this disclosure should be determined by the appended claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: The text data to be synthesized is obtained, and language recognition is performed on the text data to determine at least one target language category to which the text data belongs; Based on the at least one target language category, the text data is subjected to text normalization processing to obtain normalized text; The unified phoneme sequence corresponding to the normalized text is input into a pre-trained shared acoustic model to obtain the target acoustic features; wherein, each unified phoneme in the unified phoneme sequence is a speech unit defined by a set of pronunciation features shared across languages; Synthetic speech is obtained based on the target acoustic features.

2. The method according to claim 1, characterized in that, The method further includes: The normalized text is converted into the original phoneme sequence corresponding to the at least one target language category; According to a preset mapping relationship, each original phoneme in the original phoneme sequence is mapped to a unified phoneme in a preset unified phoneme set to obtain the unified phoneme sequence; wherein, the mapping relationship is established based on at least one of the following: place of articulation, manner of articulation, voicing attribute, and tone attribute.

3. The method according to claim 2, characterized in that, The method further includes: Training samples are input into a pre-defined teacher model to obtain reference acoustic features; wherein the teacher model is an acoustic model trained based on the unified phoneme set, and the number of parameters of the teacher model is greater than that of the pre-trained shared acoustic model; Using the training samples and the reference acoustic features as supervision signals, the initial shared acoustic model is trained to obtain the pre-trained shared acoustic model.

4. The method according to claim 1, characterized in that, The pre-trained shared acoustic model includes a shared coding network and at least one language adaptation module; The unified phoneme sequence corresponding to the normalized text is input into a pre-trained shared acoustic model to obtain the target acoustic features, including: The unified phoneme sequence is propagated layer by layer between the network layers of the shared coding network. During the propagation process, a language adaptation module corresponding to the at least one target language category is invoked to perform feature transformation on the intermediate features output by the current network layer, and finally output the target acoustic features.

5. The method according to claim 1, characterized in that, The method further includes: Predict the acoustic change information corresponding to the target acoustic features; the acoustic change information includes at least the duration information, fundamental frequency information, and energy information of each unified phoneme in the unified phoneme sequence; The target acoustic features are processed based on the acoustic change information to adjust the prosodic features of the synthesized speech; the prosodic features include at least speech rate and intonation.

6. The method according to claim 1, characterized in that, The process of synthesizing and outputting synthesized speech based on the target acoustic features includes: The target acoustic features are segmented to generate corresponding speech waveform blocks; Adjacent speech waveform blocks are spliced ​​together to generate a continuous speech waveform, which is then output as the synthesized speech.

7. The method according to claim 1, characterized in that, The method further includes: Obtain the reference speech of the target object, and extract the timbre feature vector from the reference speech; The target acoustic features are adjusted based on the timbre feature vector to determine the timbre of the synthesized speech.

8. A speech synthesis device, characterized in that, The device includes: The recognition module is used to acquire the text data to be synthesized, perform language recognition on the text data, and determine the at least one target language category to which the text data belongs; The normalization module is used to perform text normalization processing on the text data based on the at least one target language category to obtain normalized text; The feature acquisition module is used to input the unified phoneme sequence corresponding to the normalized text into a pre-trained shared acoustic model to obtain target acoustic features; wherein, each unified phoneme in the unified phoneme sequence is a speech unit defined by a set of cross-language shared pronunciation features; A synthesis module is used to obtain synthesized speech based on the target acoustic features.

9. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images