Voice duration prediction method and device, equipment and storage medium
By integrating text length and emotional features into speech duration prediction, global adaptive control of pronunciation prosody is achieved, solving the problems of uneven speech rate and insufficient emotional expression in existing technologies, and improving the naturalness and emotional expressiveness of speech synthesis.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- PING AN TECH (SHENZHEN) CO LTD
- Filing Date
- 2026-01-14
- Publication Date
- 2026-05-12
AI Technical Summary
Existing speech duration prediction methods fail to effectively consider the impact of emotional factors and text length on speech rate, resulting in a tendency for predicted durations to be averaged out. This leads to problems such as long texts being read in a drawn-out and lengthy manner, short texts being read in a rapid and abrupt manner, and emotional expression being bland.
By fusing text length features and sentiment features at the input of the duration predictor, a multimodal feature fusion mechanism is adopted. Text length features are introduced to perceive the macroscopic scale of sentences, and the prosodic distortion patterns under different emotional states are learned through sentiment features, so as to achieve global adaptive control of pronunciation prosody.
It significantly improves the accuracy of duration prediction and the naturalness of synthesized speech, ensuring the natural fluency and human-like quality of speech synthesis in terms of rhythm and emotional expression.
Smart Images
Figure CN122024698A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for predicting speech duration. Background Technology
[0002] As a crucial interface for human-computer interaction, the naturalness of text-to-speech (TTS) systems largely depends on the accuracy of prosody generation. The duration prediction module plays a central role, its task being to accurately predict the duration of each phoneme or articulation unit in the text over time, providing an alignment benchmark for subsequent acoustic feature generation.
[0003] However, most existing duration prediction methods rely solely on the local context or statistical regularities of the phoneme sequence itself for inference, resulting in a tendency for the predicted duration to be averaged out. This traditional approach suffers from a significant technical flaw: in real-world human expression, speech rate is not constant. First, emotions significantly alter speech duration; for example, anger typically corresponds to shorter phoneme durations (faster speech rate), while sadness corresponds to longer phoneme durations (slower speech rate). Second, text length also macroscopically impacts prosody; to maintain a breathing rhythm, the average phoneme duration in long sentences is usually compressed, while short sentences are stretched. Therefore, the predicted durations output by existing models tend to be averaged out, leading to biases in duration prediction. Summary of the Invention
[0004] This application provides a method, apparatus, computer device, and computer-readable storage medium for predicting voice duration, aiming to improve the problem of duration prediction deviation in traditional solutions.
[0005] A speech duration prediction method, the method comprising: Obtain the text sequence of the text to be synthesized, and extract the text encoding features based on the text sequence using a text encoder; The target sentiment information of the text to be synthesized is obtained, and sentiment features are extracted based on the target sentiment information using a sentiment encoder; The text length of the text to be synthesized is obtained, and the length feature is extracted based on the text length using the length feature extraction module; The text encoding features, the sentiment features, and the length features are input into a pre-built duration predictor, which outputs the predicted duration corresponding to each phoneme unit in the text sequence.
[0006] In one implementation, obtaining the text length of the text to be synthesized includes: Count the number of phonemes or characters contained in the text to be synthesized; The number of phonemes or the number of characters is determined as the text length of the text to be synthesized.
[0007] In one implementation, obtaining the target sentiment information of the text to be synthesized includes: Obtain the preset sentiment tags or sentiment embedding vectors for the text to be synthesized; The target emotional information is constructed by the emotional encoder based on the emotional tag or the emotional embedding vector.
[0008] In one implementation, the duration predictor is trained in the following manner: Obtain a training sample set, which includes sample text sequences, sample lengths, sample sentiment features, and corresponding real durations. The sample encoding features corresponding to the sample text sequence, the sample sentiment features, and the sample length are input into the duration prediction model to be trained to obtain the predicted duration of the sample. Calculate the loss value between the predicted duration of the sample and the actual duration; The parameters of the duration prediction model to be trained are updated based on the loss value until a preset training stopping condition is met.
[0009] In one implementation, calculating the loss value between the predicted sample duration and the actual duration includes: The difference between the predicted duration of the sample and the actual duration is calculated using the L1 loss function or the L2 loss function to obtain the loss value.
[0010] In one implementation, after obtaining the prediction duration corresponding to each phoneme unit, the method further includes: The predicted duration and the text encoding features are input into the acoustic model, and the acoustic model generates the corresponding Mel spectrum. The Mel spectrum is input into a vocoder for synthesis processing, and synthesized speech is output.
[0011] In one implementation, the method further includes: Extract the complexity features of the text to be synthesized; The number of sampling steps of the acoustic model in generating the Mel spectrum is dynamically determined based on the length feature and the complexity feature. The number of sampling steps is positively correlated with the text length represented by the length feature and the text complexity represented by the complexity feature.
[0012] In one implementation, inputting the text encoding features, the sentiment features, and the length features into a pre-constructed duration predictor includes: The duration predictor adjusts the overall speech rate prosody of the text sequence based on the length feature; When the length feature is greater than the first threshold, the predicted duration of a single phoneme unit output by the duration predictor is shortened to make the generated speech prosody more compact. When the length feature is less than the second threshold, the predicted duration of a single phoneme unit output by the duration predictor is increased to make the generated speech prosody more relaxed.
[0013] A speech duration prediction device, the device comprising: The text feature extraction module is used to obtain the text sequence of the text to be synthesized, and extract text encoding features based on the text sequence through a text encoder; The sentiment feature extraction module is used to obtain the target sentiment information of the text to be synthesized, and extract sentiment features based on the target sentiment information through a sentiment encoder; The length feature extraction module is used to obtain the text length of the text to be synthesized, and extract length features based on the text length; The duration prediction module is used to input the text encoding features, the sentiment features, and the length features into a pre-constructed duration predictor, which outputs the predicted duration corresponding to each phoneme unit in the text sequence.
[0014] A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the computer program, implements the speech duration prediction method as described in any of the preceding claims.
[0015] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.
[0016] The technical solution provided in this application provides a speech duration prediction method. This embodiment innovatively integrates text length features and emotional features at the input of the duration predictor, achieving global adaptive control of pronunciation prosody. At the principle level, introducing text length features enables the model to perceive the macroscopic scale of sentences, thereby establishing a dynamic mapping mechanism between text length and average speech rate, simulating the physiological isochronic tendency of humans to read long sentences quickly (compressing phoneme duration) and short sentences slowly (stretching phoneme duration) during reading aloud. Introducing emotional features as a conditional bias allows the model to learn and reproduce the unique prosodic distortion patterns under different emotional states. This multimodal feature fusion mechanism breaks the limitation of traditional methods that rely solely on local context leading to convergent predictions. The output predicted duration is no longer a rigid statistical average, but a dynamic value that can be adjusted in real time according to the scale of the context and the emotional tone. This effectively solves the core technical problems of long text reading being tedious and verbose, short text reading being abrupt and abrupt, and emotional expression being bland in existing technologies, significantly improving the accuracy of duration prediction and the naturalness and anthropomorphism of subsequent synthesized speech. Attached Figure Description
[0017] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0018] Figure 1 This is a processing architecture diagram of a speech duration prediction method provided in an embodiment of this application; Figure 2 This is a flowchart of a speech duration prediction method provided in an embodiment of this application; Figure 3 This is a structural diagram of a speech duration prediction device provided in an embodiment of this application; Figure 4 This is a structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation
[0019] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] Speech synthesis systems typically rely on duration prediction modules to predict the pronunciation duration of each phoneme or word in a text, guiding the acoustic model to generate natural and fluent speech. Existing duration prediction methods are mostly based on phoneme encoding, contextual information, or statistical patterns, but rarely consider the impact of emotional factors and text length on speech rate. In real speech, different emotions significantly alter speech rate and prosody; for example, anger or excitement is expressed faster, while sadness or hesitation is expressed slower. Simultaneously, text length also affects speech rate; long sentences may lead to localized changes in speech rate, while short sentences may exhibit excessively fast rhythms or unnatural pauses. Because traditional duration prediction methods ignore these factors, generated speech often deviates in rhythm and prosody, manifesting as uneven speech rate and unreasonable pauses, resulting in insufficient emotional expression. Therefore, how to fully consider emotional and text length information during duration prediction to improve the naturalness and emotional expressiveness of generated speech has become a pressing issue for current speech synthesis technologies.
[0021] like Figure 1 As shown, Figure 1 This is a schematic diagram of a speech duration prediction processing framework provided in this application embodiment. The method involves: acquiring a text sequence of the text to be synthesized; extracting text encoding features based on the text sequence using a text encoder; acquiring target sentiment information of the text to be synthesized; extracting sentiment features based on the target sentiment information using a sentiment encoder; acquiring the text length of the text to be synthesized; and extracting length features based on the text length using a length feature extraction module. The text encoding features, sentiment features, and length features are input into a pre-built duration predictor, which outputs the predicted duration corresponding to each phoneme unit in the text sequence. This predicted duration can then be used for speech synthesis, animation, subtitle alignment, and other processing. This method can meet various application needs. For example, in the fintech field, it can be used for speech synthesis in customer service assistants for financial businesses such as insurance; in the medical technology field, it can be used for processing in smart terminals in medical services; or in other speech processing applications, the specific implementation of this application embodiment is not limited; or in other intelligent voice dialogue scenarios, the implementation of this application embodiment is not limited.
[0022] It should be noted that the above method can be implemented through a server, which can be a standalone server or a server cluster consisting of multiple servers, without any specific limitation.
[0023] This application provides a method for predicting speech duration. Please refer to the following embodiments. Figure 2 This includes the following steps: S10: Obtain the text sequence of the text to be synthesized, and extract the text encoding features based on the text sequence using a text encoder; S20: Obtain the target emotion information of the text to be synthesized, and extract emotion features based on the target emotion information through an emotion encoder; S30: Obtain the text length of the text to be synthesized, and extract length features based on the text length through a length feature extraction module; S40: Input the text encoding feature, the emotion feature, and the length feature into a pre-constructed duration predictor, and the duration predictor outputs the predicted duration corresponding to each phoneme unit in the text sequence.
[0024] In this embodiment, the text sequence x = {x1, x2,..., xT} refers to the list of standardized language symbols generated after the text to be synthesized undergoes front-end processing (Frontend). Specifically, the system first performs text cleaning and word segmentation on the text to be synthesized, and then converts Chinese characters or words into a phoneme ID sequence representing the actual pronunciation through a Grapheme-to-Phoneme (G2P) module. For example, for the input text "你好", its text sequence may be represented as / n / / i / / h / / a / / o / . Subsequently, this sequence is input into the text encoder. The encoder preferably adopts a deep neural network architecture based on Transformer or Conformer, and uses the self-attention mechanism to capture the long-distance dependencies between phonemes, mapping the discrete phoneme IDs into dense high-dimensional vectors, that is, text encoding features. This feature contains the semantic attributes and context information of phonemes, providing a basic linguistic reference for the pronunciation duration.
[0025] Meanwhile, the system obtains the target emotion information, which can be the discrete emotion label e input by the user (such as sadness, anger), or an audio embedding h as a style reference. emo These information are mapped into emotion features through an emotion encoder (such as a lookup table Embedding Layer or a style encoder). Physically, this feature is a hidden vector that represents the degree of distortion of a specific emotion on the pronunciation rhythm in a high-dimensional space. For example, the angry emotion usually corresponds to a feature distribution with high energy and short duration, while the sad emotion corresponds to a feature distribution with low energy and long drawn-out sounds. In addition, the system also quantifies the text length by counting the number of phonemes or characters and converts it into a length feature through a length feature extraction module (such as position encoding or a linear projection layer).
[0026] Finally, the Duration Predictor (DP), as the core component, receives the fused input of the three features mentioned above. For example, this predictor is typically composed of a multi-layer one-dimensional convolutional neural network (1D-CNN), which predicts the frame number or time value corresponding to each phoneme through nonlinear transformation regression. The duration predictor outputs the predicted duration corresponding to each phoneme unit in the text sequence.
[0027] As can be seen, this embodiment provides a speech duration prediction method. This embodiment innovatively integrates text length features and sentiment features at the input of the duration predictor, achieving global adaptive control of pronunciation prosody. At the principle level, introducing text length features allows the model to perceive the macroscopic scale of sentences, thereby establishing a dynamic mapping mechanism between text length and average speech rate, simulating the physiological isochronic tendency of humans to read long sentences quickly (compressing phoneme duration) and short sentences slowly (stretching phoneme duration) during reading aloud. Introducing sentiment features serves as a conditional bias, enabling the model to learn and reproduce the unique prosodic distortion patterns under different emotional states. This multimodal feature fusion mechanism breaks the limitation of traditional methods that rely solely on local context leading to convergent predictions, making the output predicted duration no longer a rigid statistical average, but a dynamic value that can be adjusted in real time according to the scale of the context and the emotional tone. This effectively solves the core technical problems of long text reading being tedious and lengthy, short text reading being abrupt and abrupt, and emotional expression being bland in existing technologies, significantly improving the accuracy of duration prediction and the naturalness and human-likeness of subsequent synthesized speech.
[0028] In one embodiment, step S30, namely obtaining the text length of the text to be synthesized, Includes the following steps: S31: Count the number of phonemes or characters contained in the text to be synthesized; S32: The number of phonemes or the number of characters is determined as the text length of the text to be synthesized.
[0029] In this embodiment, the system directly performs statistical operations during the text preprocessing stage. The number of phonemes here refers to the total number of the smallest articulatory units constituting speech (e.g., "hello" contains 4 phonemes), while the number of characters is the number of words in the original text. This method of acquisition is simple and direct, effectively representing the physical scale of the input text. The principle is that the number of phonemes directly corresponds to the amount of spoken content, making it the most objective physical indicator of sentence length. Using this indicator as input helps the model establish a length-speech rate correlation mechanism. For example, for a long text containing 50 phonemes, the model learns to appropriately compress the average duration of each phoneme to maintain coherence; while for a short text containing 3 phonemes, the model learns to appropriately extend the duration to enhance the tone.
[0030] In this embodiment, by introducing a specific number of phonemes or characters as a length feature, a low-computational-cost but highly relevant global constraint variable is provided to the model. This enables the model to quickly determine the length attribute of sentences based on simple statistical values, thereby accurately performing macro-level speech rate adjustment and effectively ensuring the consistency and rationality of speech rate for sentences of varying lengths.
[0031] In one embodiment, step S20, obtaining the target sentiment information of the text to be synthesized, includes the following steps: S21: Obtain the preset sentiment tags or sentiment embedding vectors for the text to be synthesized; S22: Construct the target emotional information based on the emotional label or the emotional embedding vector using the emotional encoder.
[0032] In this embodiment, if the input is a discrete sentiment label (e.g., Label=1 represents happiness, Label=2 represents sadness), the sentiment encoder can map it to a vector using a lookup table; if the input is a continuous sentiment embedding vector (e.g., sentiment analysis output from an upstream dialogue system), it can be projected onto the target dimension through a linear layer. This design allows the system to flexibly adapt to different sentiment control methods. In practical applications, this is suitable for role-playing in audiobooks or real-time interaction with virtual digital humans.
[0033] As can be seen, in this embodiment, by explicitly constructing and inputting target emotional information, the speech synthesis system is endowed with controllable emotional expressiveness. The model can learn specific prosodic patterns in different emotional states (e.g., faster speech rate when angry, slower speech rate when sad), thereby accurately conveying the expected emotional tone when generating speech, enhancing the persuasiveness of voice interaction and user experience.
[0034] In one embodiment, model training is required to enable the model to possess the aforementioned capabilities. The duration predictor is trained in the following manner: S101: Obtain a training sample set, which includes sample text sequences, sample lengths, sample sentiment features, and corresponding real durations; S102: Input the sample encoding features corresponding to the sample text sequence, the sample sentiment features, and the sample length into the duration prediction model to be trained to obtain the predicted duration of the sample; S103: Calculate the loss value between the predicted duration of the sample and the actual duration; S104: Update the parameters of the duration prediction model to be trained based on the loss value until the preset training stop condition is met.
[0035] In this embodiment, a supervised learning framework is employed. Training data typically contains quadruples. (x,e,L,d real ), where the actual duration d real It is extracted from real recordings using a forced alignment algorithm (such as MontrealForcedAligner) and represents the actual duration of each phoneme in natural human pronunciation. During training, the model simultaneously receives text, sentiment, and length information as conditions to fit the true duration distribution. The backpropagation algorithm continuously adjusts the model parameters based on the prediction error until the model converges.
[0036] In this embodiment, by introducing length and sentiment as explicit supervision signals during the training phase, the model is forced to learn the nonlinear mapping relationship between these global variables and local duration. Compared to traditional training methods that only input text, this multi-condition training strategy can significantly reduce the model's prediction error and endow the model with stronger generalization ability and robustness when faced with unseen long texts or specific sentiments during the inference phase.
[0037] In one embodiment, step S103, namely calculating the loss value between the predicted sample duration and the actual duration, includes: The difference between the predicted duration of the sample and the actual duration is calculated using the L1 loss function or the L2 loss function to obtain the loss value.
[0038] Based on the above characteristics, in this embodiment, the system can calculate the loss using the following formula: or; It is the total number of phonemes in the sequence. It is the predicted duration. This refers to the actual duration. To ensure numerical stability and avoid gradient explosion, calculations are typically performed in the log-domain. L1 loss (mean absolute error) is insensitive to outliers, allowing the model to focus more on the overall trend; while L2 loss (mean squared error) penalizes large errors more severely and converges faster.
[0039] In this embodiment, the difference between the predicted duration and the actual duration is directly penalized by using an L1 or L2 loss function, strictly adhering to the optimization direction of the regression task. This ensures that the duration trajectory predicted by the model approximates the prosodic curve of the real speech as closely as possible, thereby effectively avoiding problems such as rhythmic breakdown and abrupt pauses in the generated speech and guaranteeing the stability of speech quality during the sampling process.
[0040] After the predicted duration is obtained, speech synthesis is usually required. In one embodiment, after step S40, that is, after obtaining the predicted duration corresponding to each phoneme unit, the method further includes the following steps: S50: Input the predicted duration and the text encoding features into the acoustic model, and the acoustic model generates the corresponding Mel spectrum; S60: Input the Mel spectrum into the vocoder for synthesis processing and output synthesized speech.
[0041] In this embodiment, the predicted duration sequence is first used by a length regulator to copy and expand the text-encoded features according to the duration, aligning them with the frame number of the target audio in the time dimension. Subsequently, an acoustic model (such as a Transformer-based decoder) generates a Mel-spectrum based on the aligned features; this is a time-frequency graph that simulates the characteristics of human hearing. Finally, a vocoder (such as HiFi-GAN) restores the Mel-spectrum to a time-domain waveform.
[0042] In this embodiment, a complete closed-loop link from duration prediction to final speech generation is constructed. By passing the accurately predicted duration information to the acoustic model and vocoder, it is ensured that the final output synthesized speech is not only accurate in content, but also natural and fluent in rhythm and prosody, achieving high-quality end-to-end speech synthesis.
[0043] To improve system efficiency while ensuring voice quality, this embodiment also specifically mentions a dynamic sampling strategy based on length and complexity. Specifically, in one embodiment, the method further includes: S105: Extract the complexity features of the text to be synthesized; S106: Dynamically determine the number of sampling steps the acoustic model takes when generating the Mel spectrum based on the length feature and the complexity feature; wherein the number of sampling steps is positively correlated with the text length represented by the length feature and the text complexity represented by the complexity feature. In this embodiment, the system analyzes the complexity features (such as the proportion of rare characters, syntax tree depth, etc.) and length features of the input text. Based on the characteristics of the diffusion model or iterative generation model, the number of sampling steps determines the balance between generation quality and speed. The strategy logic is as follows: for short or simple text, due to the low information density and low generation difficulty, the model allocates fewer sampling steps (e.g., 10 steps) to quickly generate high-quality speech; for long or complex text, in order to prevent spectral blurring or unclear pronunciation caused by oversimplification of the generation process, the model automatically allocates more sampling steps (e.g., 50 steps) to refine the details.
[0044] This embodiment implements an intelligent mechanism for on-demand allocation of computing resources. By dynamically adjusting the number of sampling steps, it can significantly improve inference speed and reduce latency and power consumption when dealing with short texts; while when dealing with long and complex sentences, it can prevent fuzzy generation or unclear pronunciation by increasing the amount of computation. This strategy effectively balances generation efficiency and speech fidelity, improving the system's adaptability in different scenarios.
[0045] Furthermore, to further optimize prosody, this application also provides a threshold-based adjustment mechanism. In one embodiment, step S40, which is the step of inputting features into the duration predictor, includes: S41: Adjust the overall speech rate prosody of the text sequence based on the length feature using the duration predictor; S42: When the length feature is greater than the first threshold, the predicted duration of a single phoneme unit output by the duration predictor is shortened to make the generated speech prosody more compact; S43: When the length feature is less than the second threshold, the predicted duration of a single phoneme unit output by the duration predictor is increased to make the generated speech prosody more relaxed.
[0046] To address the aforementioned characteristics, this embodiment utilizes length features to resolve the issue of inconsistent speech rates between sentences of varying lengths. The system sets a first threshold (e.g., 30 phonemes) and a second threshold (e.g., 5 phonemes). When a text length L > 30 is detected, the model automatically applies a scaling factor less than 1 to shorten the duration of the output phonemes, thereby accelerating the speech rate and creating a sense of urgency. When a text length L < 5 is detected, the model applies a scaling factor greater than 1 to increase the duration of the phonemes, thereby slowing down the speech rate and creating a sense of relaxation.
[0047] In this embodiment, the threshold-based adaptive adjustment mechanism simulates the human physiological habit of reading long sentences quickly and short sentences slowly (i.e., isochronism). This effectively avoids the unnatural feeling of long sentences being read slowly and tediously, and short sentences being read abruptly and abruptly, which is common in traditional fixed-speed methods, thus significantly improving the listening comfort and rhythmic rationality of the synthesized speech.
[0048] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.
[0049] This application also provides a speech synthesis system based on multi-condition fusion, which corresponds one-to-one with a speech synthesis method described in the above embodiments. Please refer to... Figure 3 ,include: The text feature extraction module 301 is used to obtain the text sequence of the text to be synthesized, and extract text encoding features based on the text sequence through a text encoder; The sentiment feature extraction module 302 is used to obtain the target sentiment information of the text to be synthesized, and extract sentiment features based on the target sentiment information through a sentiment encoder; The length feature extraction module 303 is used to obtain the text length of the text to be synthesized, and extract length features based on the text length; The duration prediction module 304 is used to input the text encoding features, the sentiment features and the length features into a pre-constructed duration predictor, and the duration predictor outputs the predicted duration corresponding to each phoneme unit in the text sequence.
[0050] In one embodiment, obtaining the text length of the text to be synthesized includes: Count the number of phonemes or characters contained in the text to be synthesized; The number of phonemes or the number of characters is determined as the text length of the text to be synthesized.
[0051] In one embodiment, obtaining the target sentiment information of the text to be synthesized includes: Obtain the preset sentiment tags or sentiment embedding vectors for the text to be synthesized; The target emotional information is constructed by the emotional encoder based on the emotional tag or the emotional embedding vector.
[0052] In one embodiment, the duration predictor is trained in the following manner: Obtain a training sample set, which includes sample text sequences, sample lengths, sample sentiment features, and corresponding real durations. The sample encoding features corresponding to the sample text sequence, the sample sentiment features, and the sample length are input into the duration prediction model to be trained to obtain the predicted duration of the sample. Calculate the loss value between the predicted duration of the sample and the actual duration; The parameters of the duration prediction model to be trained are updated based on the loss value until a preset training stopping condition is met.
[0053] In one embodiment, calculating the loss value between the predicted sample duration and the actual duration includes: The difference between the predicted duration of the sample and the actual duration is calculated using the L1 loss function or the L2 loss function to obtain the loss value.
[0054] In one embodiment, after obtaining the prediction duration corresponding to each phoneme unit, the method further includes: The predicted duration and the text encoding features are input into the acoustic model, and the acoustic model generates the corresponding Mel spectrum. The Mel spectrum is input into a vocoder for synthesis processing, and synthesized speech is output.
[0055] In one embodiment, the device is also used for: Extract the complexity features of the text to be synthesized; The number of sampling steps of the acoustic model in generating the Mel spectrum is dynamically determined based on the length feature and the complexity feature. The number of sampling steps is positively correlated with the text length represented by the length feature and the text complexity represented by the complexity feature.
[0056] In one embodiment, inputting the text encoding features, the sentiment features, and the length features into a pre-constructed duration predictor includes: The duration predictor adjusts the overall speech rate prosody of the text sequence based on the length feature; When the length feature is greater than the first threshold, the predicted duration of a single phoneme unit output by the duration predictor is shortened to make the generated speech prosody more compact. When the length feature is less than the second threshold, the predicted duration of a single phoneme unit output by the duration predictor is increased to make the generated speech prosody more relaxed.
[0057] As can be seen, this embodiment provides a speech duration prediction device and a speech duration prediction method. This embodiment innovatively integrates text length features and sentiment features at the input of the duration predictor, achieving global adaptive control of pronunciation prosody. At the principle level, introducing text length features enables the model to perceive the macroscopic scale of sentences, thereby establishing a dynamic mapping mechanism between text length and average speech rate, simulating the physiological isochronic tendency of humans to read long sentences quickly (compressing phoneme duration) and short sentences slowly (stretching phoneme duration) during reading aloud. Introducing sentiment features serves as a conditional bias, enabling the model to learn and reproduce the unique prosodic distortion patterns under different emotional states. This multimodal feature fusion mechanism breaks the limitation of traditional methods that rely solely on local context leading to convergent predictions, making the output predicted duration no longer a rigid statistical average, but a dynamic value that can be adjusted in real time according to the scale of context and emotional tone. This effectively solves the core technical problems of long text reading being tedious and lengthy, short text reading being abrupt and abrupt, and emotional expression being bland in existing technologies, significantly improving the accuracy of duration prediction and the naturalness and human-likeness of subsequent synthesized speech.
[0058] For specific limitations regarding the speech duration prediction device, please refer to the limitations of the speech duration prediction method above, which will not be repeated here. Each module in the aforementioned speech duration prediction device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.
[0059] In one embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 4 As shown. The computer device includes a processor, memory, network interface, and database connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. When executed by the processor, the computer program implements the functions of a speech duration prediction device, or, when executed, implements the steps of a speech duration prediction method.
[0060] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the function of a speech duration prediction device, or executes the steps of a speech duration prediction method, to avoid repetition.
[0061] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When executed by a processor, the computer program implements the functions of a speech synthesis system, or when executed, implements the steps of a speech synthesis method. To avoid repetition, these steps will not be repeated here.
[0062] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.
[0063] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.
[0064] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.
Claims
1. A method for predicting speech duration, characterized in that, The method includes: Obtain the text sequence of the text to be synthesized, and extract the text encoding features based on the text sequence using a text encoder; The target sentiment information of the text to be synthesized is obtained, and sentiment features are extracted based on the target sentiment information using a sentiment encoder; The text length of the text to be synthesized is obtained, and the length feature is extracted based on the text length using the length feature extraction module; The text encoding features, the sentiment features, and the length features are input into a pre-built duration predictor, which outputs the predicted duration corresponding to each phoneme unit in the text sequence.
2. The method according to claim 1, characterized in that, The step of obtaining the text length of the text to be synthesized includes: Count the number of phonemes or characters contained in the text to be synthesized; The number of phonemes or the number of characters is determined as the text length of the text to be synthesized.
3. The method according to claim 1, characterized in that, The step of obtaining the target sentiment information of the text to be synthesized includes: Obtain the preset sentiment tags or sentiment embedding vectors for the text to be synthesized; The target emotional information is constructed by the emotional encoder based on the emotional tag or the emotional embedding vector.
4. The method according to claim 1, characterized in that, The duration predictor is trained in the following way: Obtain a training sample set, which includes sample text sequences, sample lengths, sample sentiment features, and corresponding real durations. The sample encoding features corresponding to the sample text sequence, the sample sentiment features, and the sample length are input into the duration prediction model to be trained to obtain the predicted duration of the sample. Calculate the loss value between the predicted duration of the sample and the actual duration; The parameters of the duration prediction model to be trained are updated based on the loss value until a preset training stopping condition is met.
5. The speech duration prediction method according to claim 1, characterized in that, After obtaining the prediction duration corresponding to each phoneme unit, the method further includes: The predicted duration and the text encoding features are input into the acoustic model, and the acoustic model generates the corresponding Mel spectrum. The Mel spectrum is input into a vocoder for synthesis processing, and synthesized speech is output.
6. The method according to claim 5, characterized in that, The method further includes: Extract the complexity features of the text to be synthesized; The number of sampling steps of the acoustic model in generating the Mel spectrum is dynamically determined based on the length feature and the complexity feature. The number of sampling steps is positively correlated with the text length represented by the length feature and the text complexity represented by the complexity feature.
7. The speech duration prediction method according to any one of claims 1-6, characterized in that, The step of inputting the text encoding features, the sentiment features, and the length features into a pre-constructed duration predictor includes: The duration predictor adjusts the overall speech rate prosody of the text sequence based on the length feature; When the length feature is greater than the first threshold, the predicted duration of a single phoneme unit output by the duration predictor is shortened to make the generated speech prosody more compact. When the length feature is less than the second threshold, the predicted duration of a single phoneme unit output by the duration predictor is increased to make the generated speech prosody more relaxed.
8. A speech duration prediction device, characterized in that, The device includes: The text feature extraction module is used to obtain the text sequence of the text to be synthesized, and extract text encoding features based on the text sequence through a text encoder; The sentiment feature extraction module is used to obtain the target sentiment information of the text to be synthesized, and extract sentiment features based on the target sentiment information through a sentiment encoder; The length feature extraction module is used to obtain the text length of the text to be synthesized, and extract length features based on the text length; The duration prediction module is used to input the text encoding features, the sentiment features, and the length features into a pre-constructed duration predictor, which outputs the predicted duration corresponding to each phoneme unit in the text sequence.
9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the speech duration prediction method as described in any one of claims 1 to 7.
10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.