Training method for speech prosody prediction model, speech prosody prediction method and device

Abstract

This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech and digital healthcare. It discloses a method for training a speech prosody prediction model, a speech prosody prediction method, and an apparatus. The method includes: constructing a speech prosody prediction model, including a prosody encoder, a first prosody decoder, and a second prosody decoder; acquiring reference audio and reference text, extracting acoustic features and target prosodic features from the reference audio, converting the reference text into a reference phoneme sequence, and aligning the acoustic features; performing synchronous masking based on the alignment relationship to obtain masked acoustic features and a masked reference phoneme sequence, and inputting these into the prosody encoder to obtain a latent prosodic embedding; inputting the latent prosodic embedding into the two decoders to predict the masked prosodic features; determining the target training loss based on the predicted prosodic features and the target prosodic features, and updating the model parameters to obtain the trained speech prosody prediction model. This method improves the generalization and naturalness of zero-shot prosodic prediction.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology and can be applied to the fields of fintech / digital healthcare, particularly to a method for training a speech prosody prediction model, a speech prosody prediction method, and a device. Background Technology

[0002] With the development of artificial intelligence, speech synthesis, and intelligent voice interaction technologies, speech prosody prediction has become a crucial technical aspect for improving speech naturalness, emotional expression, and interactive experience. Existing prosody modeling methods mainly fall into two categories: one is an embedded modeling method that integrates the prosody prediction module into an end-to-end text-to-speech system, and the other is a method that uses a pre-trained model for fine-tuning in downstream tasks. While the aforementioned technologies have been applied to some extent in general speech generation tasks, they still have significant limitations: On the one hand, existing prosodic models are usually deeply coupled with specific speech synthesis architectures, making it difficult to adapt them as independent modules to different downstream systems, resulting in poor versatility and transferability; on the other hand, when faced with speech generation needs that do not involve speakers, styles, or cross-scene conditions, existing solutions usually still rely on retraining or fine-tuning, making it difficult to meet the prosodic transfer requirements under zero-shot conditions; in addition, some solutions mainly rely on extracting style information from reference audio, failing to fully utilize the prosodic cues such as sentence structure, semantic stress, and pause patterns contained in the corresponding text of the reference audio, thus affecting the accuracy of prosodic prediction; at the same time, when processing variable-length audio sequences and text sequences, traditional models also suffer from high computational complexity and insufficient ability to represent high-level prosodic patterns of fixed length. Especially in application scenarios with high requirements for the quality of intelligent voice interaction, such as financial fintech and digital healthcare, such as financial digital human broadcasting, intelligent customer service voice response, medical triage voice interaction, and health education voice generation, the system is often required to quickly transfer speaking style based on a small number of reference voices, and to generate natural, stable and context-appropriate prosodic output for different text content. However, existing technologies still have shortcomings in terms of versatility, zero-sample capability, text and acoustic information collaborative modeling capability, and robustness in complex scenarios. Summary of the Invention

[0003] The main technical problems addressed by the embodiments of this application are the insufficient zero-sample generalization ability, inadequate collaborative utilization of reference audio and text information, and low naturalness of prosody prediction in existing speech prosody prediction technologies.

[0004] To address the aforementioned technical problems, the first technical solution adopted in this application is: providing a method for training a speech prosody prediction model, comprising: constructing a speech prosody prediction model, the speech prosody prediction model including a prosody encoder, a first prosody decoder, and a second prosody decoder; acquiring reference audio and reference text, extracting acoustic features of the reference audio and target prosodic features corresponding to the reference audio, converting the reference text into a reference phoneme sequence, and aligning the acoustic features and the reference phoneme sequence; based on the alignment relationship between the acoustic features and the reference phoneme sequence, performing masking on the features to be masked in the acoustic features and the features to be masked in the reference phoneme sequence. The reference phonemes are synchronously masked to obtain masked acoustic features and a masked reference phoneme sequence. The masked acoustic features and the masked reference phoneme sequence are input into the prosodic encoder to encode and compress the variable-length bimodal input, resulting in a fixed-length latent prosodic embedding. The latent prosodic embedding is input into the first prosodic decoder and the second prosodic decoder to predict the prosodic features corresponding to the masked parts based on the masked prosodic input and the masked text, respectively. The target training loss is determined based on the predicted prosodic features and the target prosodic features, and the parameters of the speech prosodic prediction model are updated according to the target training loss to obtain the trained speech prosodic prediction model.

[0005] To address the aforementioned technical problems, the second technical solution adopted in this application is: providing a speech prosody prediction method, comprising: acquiring reference audio, reference text, and target text; extracting acoustic features from the reference audio, converting the reference text into a reference phoneme sequence, and aligning the acoustic features and the reference phoneme sequence; inputting the acoustic features and the reference phoneme sequence into the prosody encoder in a trained speech prosody prediction model to obtain a fixed-length latent prosody embedding; inputting the target text and the latent prosody embedding into the prosody decoder in the trained speech prosody prediction model to obtain the prosody features corresponding to the target text; and outputting the prosody features as the speech prosody prediction result corresponding to the target text; wherein the trained speech prosody prediction model is trained using the aforementioned speech prosody prediction model training method.

[0006] To address the aforementioned technical problems, the third technical solution adopted in this application is: providing a speech prosody prediction model training device, comprising: a model building module for building a speech prosody prediction model, the speech prosody prediction model including a prosody encoder, a first prosody decoder, and a second prosody decoder; a data acquisition and processing module for acquiring reference audio and reference text, extracting acoustic features of the reference audio and target prosodic features corresponding to the reference audio, converting the reference text into a reference phoneme sequence, and aligning the acoustic features and the reference phoneme sequence; and a synchronization masking module for synchronizing the features to be masked in the acoustic features and the features to be masked in the reference phoneme sequence based on the alignment relationship between the acoustic features and the reference phoneme sequence. The system performs synchronous masking on reference phonemes to obtain masked acoustic features and a masked reference phoneme sequence. An encoding and compression module inputs the masked acoustic features and masked reference phoneme sequence into a prosodic encoder to encode and compress the variable-length bimodal input, resulting in a fixed-length latent prosodic embedding. A dual-decoding prediction module inputs the latent prosodic embedding into a first and second prosodic decoder, predicting the prosodic features corresponding to the masked portion based on the masked prosodic input and the masked text, respectively. A loss calculation and update module determines the target training loss based on the predicted prosodic features and the target prosodic features, and updates the parameters of the speech prosodic prediction model according to the target training loss, resulting in the trained speech prosodic prediction model.

[0007] To solve the above-mentioned technical problems, the fourth technical solution adopted in the embodiments of this application is: to provide an electronic device, including: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the speech prosody prediction model training method or the speech prosody prediction method as described above.

[0008] To solve the above-mentioned technical problems, the fifth technical solution adopted in the embodiments of this application is: to provide a non-volatile computer-readable storage medium, wherein the non-volatile computer-readable storage medium stores computer-executable instructions, and when the computer-executable instructions are executed by an electronic device, the electronic device performs the speech prosody prediction model training method or the speech prosody prediction method as described above.

[0009] Unlike related technologies, this application introduces a prosodic encoder, dual decoding paths, and conditional modulation structures into the same speech prosodic prediction model, forming a processing chain within the model from style extraction and dual-path constraints to fine-grained control. Acoustic information from the reference speech and textual information from the reference text are jointly modeled in the encoding stage and compressed into a fixed-length latent prosodic embedding. This improves the stability of processing variable-length bimodal inputs while obtaining a more stable overall prosodic style representation. Based on this, the first decoding path constrains masked prosodic recovery, and the second decoding path constrains prosodic prediction under textual conditions, enabling the latent prosodic embedding to possess both acoustic prosodic carrying capacity and textual prosodic mapping capability. Furthermore, the second decoding path introduces a time-varying modulation mechanism through a conditional transformer, allowing the overall style conditions to act on feature changes at different time points. Therefore, while maintaining the overall style consistency of the reference speech, it enhances the expressive power of local prosodic changes. Based on the above model structure and training mechanism, the training phase can improve the stability and zero-shot generalization ability of dual-modal joint modeling, while the prediction phase can generate more natural, stable and stylistically consistent prosodic features for the target text without retraining or fine-tuning. Attached Figure Description

[0010] One or more embodiments are illustrated by way of example with reference to the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements having the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.

[0011] Figure 1 This is a schematic diagram of the operating environment for the speech prosody prediction model training method provided in the embodiments of this application.

[0012] Figure 2 This is a schematic diagram of the execution flow of the speech prosody prediction model training method provided in the embodiments of this application.

[0013] Figure 3 This is a schematic diagram of the execution flow of predicting prosodic features in the speech prosodic prediction model training method provided in the embodiments of this application.

[0014] Figure 4 This is a schematic diagram of the execution flow of the speech prosody prediction method provided in the embodiments of this application.

[0015] Figure 5 This is a schematic diagram of the system structure of the speech prosody prediction model training device provided in the embodiments of this application.

[0016] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a speech prosody prediction model training method or a speech prosody prediction method, as provided in the embodiments of this application. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. Software tools, components, or servers not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0018] It should be noted that, unless otherwise specified, the various features in the embodiments of this application can be combined with each other, all of which are within the protection scope of this application. Furthermore, although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device schematic diagram or the order in the flowchart.

[0019] Unless otherwise defined, all technical and scientific terms used in this specification have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in this specification is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. The term "and / or" as used in this specification includes any and all combinations of one or more of the associated listed items.

[0020] To facilitate understanding of this embodiment, a detailed description of the speech prosody prediction model training method disclosed in this application embodiment will be provided first. Please refer to [link to relevant documentation]. Figure 1 , Figure 1 This is a schematic diagram of the operating environment for the speech prosody prediction model training method provided in the embodiments of this application, such as... Figure 1 As shown, the execution subject of the speech prosody prediction model training method provided in this application embodiment is generally an electronic device with a certain computing power, such as a computer device. In some possible implementations, this speech prosody prediction model training method can be implemented by a processor calling computer-readable instructions stored in memory. Figure 1 The computer equipment mentioned can be a server. A server can be a standalone server or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. This can be understood as... Figure 1 The number of computer devices shown is merely illustrative and can be expanded in any number according to actual needs.

[0021] Please continue reading. Figure 2 , Figure 2This is a schematic diagram of the execution flow of the speech prosody prediction model training method provided in the embodiments of this application, as shown below. Figure 2 As shown, the process includes steps S11 to S16.

[0022] S11. Construct a speech prosody prediction model, which includes a prosody encoder, a first prosody decoder, and a second prosody decoder.

[0023] In step S11, a speech prosody prediction model is first constructed. This model employs an encoder-decoder architecture, comprising a prosody encoder, a first prosody decoder, and a second prosody decoder. The prosody encoder jointly models the acoustic features corresponding to the reference audio and the phoneme sequence corresponding to the reference text, extracting latent prosodic embeddings that characterize speaking style and prosodic patterns from the acoustic and textual information. The first and second prosodic decoders are connected to the prosody encoder and are used to receive the latent prosodic embeddings and predict the prosodic features corresponding to the masked portion from different input paths.

[0024] Specifically, the prosodic encoder can be understood as a prosodic information extraction module, similar to compressing variable-length reference audio and reference text into a fixed-length high-level prosodic representation. The first prosodic decoder receives the latent prosodic embedding and the masked prosodic input, and predicts the prosodic features of the masked portion based on this input, so as to learn a stable acoustic prosodic representation by constraining the prosodic encoder through the corresponding loss. The second prosodic decoder receives the latent prosodic embedding and the masked text, and recovers the prosodic features of the masked portion based on the text content and the latent prosodic representation, thereby improving the model's ability to utilize text prosodic cues. The prosodic encoder is mainly responsible for extracting and compressing prosodic information, the first prosodic decoder is mainly responsible for strengthening the training of acoustic prosodic representations, and the second prosodic decoder is mainly responsible for text-conditional prosodic prediction. Together, these three components constitute the core model structure for zero-shot speech prosodic prediction.

[0025] As an example, in fintech scenarios, historical reference audio and corresponding text messages from financial digital human customer service can be input into the model. A prosodic encoder extracts latent prosodic embeddings of the customer service broadcast style, which are then combined with risk warning text, product explanation text, or outbound call response text. A second prosodic decoder generates prosodic features that better match the tone of financial services. In digital healthcare scenarios, triage broadcasts, health education voices, or caregiver voices can be used as reference audio, and the corresponding text can be input into the model. A prosodic encoder extracts stable, friendly, or suggestive latent prosodic embeddings, which are then combined with target consultation text, medication reminder text, or rehabilitation guidance text to generate corresponding prosodic features. This model structure design not only avoids the problem of excessive reliance on single text or acoustic information in existing speech prosodic modeling technologies but also provides more natural and stable prosodic control capabilities for voice broadcasting, digital human interaction, and intelligent voice services in fintech and digital healthcare scenarios.

[0026] S12. Obtain reference audio and reference text, extract the acoustic features of the reference audio and the target prosodic features corresponding to the reference audio, convert the reference text into a reference phoneme sequence, and align the acoustic features and the reference phoneme sequence.

[0027] In step S12, reference audio and reference text are first acquired, and feature extraction processing is performed on the reference audio to obtain acoustic features that can characterize speech prosodic patterns and target prosodic features corresponding to the reference audio. Simultaneously, the reference text is converted into a reference phoneme sequence, and the acoustic features are further aligned with the reference phoneme sequence. The purpose of this step is to establish a unified data representation foundation for subsequent dual-modal joint training.

[0028] Specifically, acoustic features can be understood as temporal speech representations extracted from reference audio, while target prosodic features can be understood as the target outputs that the model needs to learn and recover during the training phase. Both originate from reference audio but have different functions: the former is more inclined to carry the original acoustic information of the reference audio, while the latter is more inclined to represent the prosodic information that needs to be predicted during training. The reference phoneme sequence obtained by converting the reference text is used to map the text content to a more suitable articulatory unit level for speech modeling. By further refining the reference text from the word level to the phoneme level, the model can more accurately capture prosodic cues related to pauses, stress, and intonation changes in the text. Furthermore, aligning acoustic features with the reference phoneme sequence can establish a correspondence between acoustic frames and text phonemes, so that subsequent synchronization masks are no longer isolated random processing of acoustic features or text features, but can achieve joint constraints of acoustics and text within the same phoneme or the same local speech segment. It not only completed the preprocessing of the reference audio and reference text, but also mapped the audio and text information that originally belonged to different modalities into a unified representation space that can correspond and be jointly modeled, creating conditions for subsequent synchronization masking at the phoneme granularity and extraction of potential prosodic embeddings through prosodic encoder.

[0029] As an example, in fintech scenarios, reference audio from financial customer service or digital human broadcasts can be used, along with the corresponding text, to extract acoustic information such as pitch, energy, and duration. Simultaneously, text content such as "risk warnings," "product explanations," and "business confirmations" is converted into phoneme sequences. The audio frames and phoneme sequences are then aligned, allowing the model to simultaneously learn the textual expression and prosodic patterns within the financial broadcast tone. In digital healthcare scenarios, reference audio from patient guidance broadcasts, health education messages, or companion prompts can be used, with the corresponding text converted into phoneme sequences. Alignment processing establishes a mapping between the acoustic segments of the speech and the text content, enabling the model to more accurately learn the prosodic features of smooth, friendly, or highly suggestive medical speech. This approach not only improves the synergy between acoustic and textual information but also provides a sufficient data foundation for subsequent masking training, fixed-length latent prosodic embedding extraction, and target text prosodic prediction.

[0030] S13. Based on the alignment relationship between acoustic features and reference phoneme sequence, perform synchronous masking processing on the features to be masked in the acoustic features and the reference phonemes to be masked in the reference phoneme sequence to obtain masked acoustic features and masked reference phoneme sequence.

[0031] As an optional implementation, the synchronization mask processing in step S13 above may also include steps S131 to S133.

[0032] S131. Determine the reference phonemes to be masked according to the phoneme dimension.

[0033] In step S131, the phoneme units to be masked are first determined in the reference phoneme sequence, and the masking granularity is limited to the phoneme level, rather than performing discrete selection directly at the acoustic frame level. This is because phonemes are the basic articulatory units in speech generation and prosodic modeling. A phoneme typically corresponds to multiple consecutive acoustic frames and maintains relatively complete local prosodic features in terms of duration, pitch trend, and energy changes. If random masking is performed directly on an acoustic frame-by-frame basis, it can easily lead to the fragmentation of consecutive acoustic segments corresponding to the same phoneme, resulting in some frames being masked and others not, thus disrupting the integrity of the phoneme-level speech structure. Therefore, pre-determining the reference phonemes to be masked according to the phoneme dimension in step S131 essentially establishes the basic processing unit for masking on the text side, enabling subsequent masking processing to revolve around the articulatory units and laying the foundation for consistency constraints between the acoustic modality and the text modality.

[0034] S132. Based on the alignment relationship, determine the acoustic features that are aligned with the reference phonemes to be masked as the features to be masked.

[0035] In step S132, the alignment relationship between the acoustic features and the reference phoneme sequence established in the previous steps is used to map the selected reference phonemes to be masked on the text side to the acoustic side, thereby determining the corresponding features to be masked. Since the reference audio is represented as continuous acoustic frames in the temporal domain, while the reference text is represented as a discrete phoneme sequence in structure, they belong to different modal data representations and therefore cannot be directly mapped. Using the alignment relationship, the time range corresponding to each reference phoneme in the reference audio and its corresponding acoustic feature segment can be clearly defined, thus accurately projecting the masking target in the text modality to the acoustic modality. Step S132 establishes the correspondence between "reference phonemes to be masked" and "acoustic features to be masked," enabling subsequent masking operations to be performed synchronously across both modalities, rather than independently processing the two modalities. This has significant technical implications for subsequent dual-modal joint representation learning.

[0036] S133. Perform synchronous masking processing on the reference phonemes to be masked and the features to be masked, so that the acoustic features corresponding to a single reference phoneme are either all masked or all unmasked.

[0037] In step S133, consistent masking constraints are applied to both the text and acoustic modalities, ensuring the masking results remain intact within phoneme boundaries. Specifically, once a reference phoneme is identified as a target for masking, all corresponding acoustic features are simultaneously identified as masked features and masked synchronously in the same processing round. Conversely, if a reference phoneme is not selected, its corresponding acoustic features remain unmasked. This synchronous masking mechanism avoids situations where a single reference phoneme is partially masked or partially unmasked on the acoustic side, reducing the risk of disrupting the internal speech structure of the phoneme. Furthermore, this mechanism ensures that the model faces bimodal incomplete inputs during training, where both "missing text articulatory units" and "missing corresponding acoustic segments" occur simultaneously. This encourages the model to learn how to recover masked prosodic features based on remaining contextual information, enhancing the collaborative modeling ability between acoustic and textual information and improving the abstraction ability of latent prosodic representations of complete prosodic patterns.

[0038] As an example, in fintech scenarios, a reference speech segment and its corresponding text from financial digital human customer service, intelligent voice outbound calls, or risk warning broadcasts can be selected as training samples. The reference text, after phoneme conversion, forms a phoneme sequence arranged chronologically, with each phoneme corresponding to a continuous acoustic frame in the reference speech. Based on pre-established alignment results, the start and end times of each phoneme, as well as the covered acoustic feature segments, can be clearly defined. During a training process, several phonemes can be selected from the phoneme sequence as masking objects for this round. Then, based on the alignment relationship, the corresponding continuous frame intervals of these phonemes in the acoustic feature sequence are located, and masking processing is simultaneously applied to both the text-side phoneme units and the corresponding acoustic frame intervals. For example, for a phoneme in business expressions such as "risk disclosure," "profit fluctuation," or "identity verification," once the phoneme is selected as a masking object, all acoustic feature frames corresponding to that phoneme participate in the masking simultaneously, rather than only masking a portion of the frames. This maintains the prosodic consistency of the financial broadcast tone at the level of complete pronunciation units. Similarly, in digital healthcare scenarios, reference speech and its corresponding text from patient guidance announcements, health education, medication reminders, or rehabilitation guidance can be selected as training samples. The correspondence between phonemes and acoustic segments can be established in the same way, and synchronous masking can be implemented at the complete phoneme unit level. For example, for target phonemes in medical prompts such as "take medication on time," "follow-up appointment time," or "examination instructions," once a target phoneme is selected, the continuous acoustic features corresponding to that phoneme are masked as a whole, while unselected phonemes and their corresponding acoustic segments are preserved. This method ensures that the information loss in the training samples remains within the scope of complete phonetic units, avoiding the fragmentation of the internal prosodic structure of individual phonemes by local acoustic frames. This facilitates the model in learning more stable bimodal prosodic recovery relationships in fintech and digital healthcare speech scenarios.

[0039] Through steps S131 to S133, the masking operation can be upgraded from random, discrete frame-level processing to a bimodal consistency processing mechanism centered around complete articulatory units. Since prosodic information is typically not independent within a single acoustic frame but rather attached to continuous patterns such as phoneme duration, local pitch variations, and energy distribution, fragmented information loss within a single phoneme can easily disrupt the continuity of the original prosodic structure, causing the model's learned loss recovery relationship to deviate from true speech patterns. By establishing synchronization constraints based on complete articulatory units, the missing articulatory information in the text modality and the corresponding missing prosodic information in the acoustic modality can maintain boundary consistency. During training, the model faces samples that are both semantically and acoustically incomplete but still structurally intact. This reduces training noise caused by cross-modal misalignment, enhances the correspondence between textual and acoustic information, and encourages the model to learn more stable prosodic recovery patterns from the remaining context. Furthermore, this processing method also helps the prosody encoder extract more holistic latent prosodic representations, improves the model's ability to abstract complete prosodic patterns, and thus provides a more reliable representation basis for subsequent mask recovery, target training loss calculation, and target text prosodic prediction in the inference stage. Ultimately, it helps to improve the stability, accuracy, and naturalness of prosodic prediction results.

[0040] S14. Input the masked acoustic features and masked reference phoneme sequence into the prosody encoder to encode and compress the variable-length bimodal input to obtain a fixed-length latent prosody embedding.

[0041] As an optional implementation, the process of obtaining the potential prosodic embedding in step S14 above may also include the following steps S141 to S144.

[0042] S141. Determine the masked acoustic features and masked reference phoneme sequence as variable-length bimodal input data, and input the variable-length bimodal input data into the prosody encoder.

[0043] In step S141, the acoustic modal data and text modal data, after synchronization masking, are organized into a unified model input format and fed into the prosodic encoder for subsequent representation learning. Since the acoustic features corresponding to the reference audio are usually in the form of time-frame sequences, while the phoneme sequences converted from the reference text are usually in the form of discrete articulator sequences, they differ in length, sampling granularity, and representation structure. Therefore, they need to be jointly determined as variable-length bimodal input data before being fed into the prosodic encoder for joint processing. The core of this step is to clarify the input object of the prosodic encoder and establish the starting point for bimodal joint encoding, so that the model no longer processes acoustic and text information separately, but instead models the correspondence, complementarity, and prosodic association between the two in a unified manner during the same encoding stage, thus providing an input basis for the subsequent extraction of potential prosodic representations.

[0044] S142. Set a fixed number of potential query vectors in the prosody encoder.

[0045] In step S142, a set of latent representation vectors with fixed length and a pre-defined number are provided to the prosodic encoder for extracting high-level prosodic representation information from variable-length bimodal input data. Since the lengths of the input reference audio and reference text can vary depending on the sample, directly generating subsequent representations using the original input length would not only increase modeling complexity but also hinder the formation of a unified style representation. Therefore, pre-setting a fixed number of latent query vectors in the prosodic encoder allows the encoding process to revolve around a fixed number of latent representation units, which actively aggregate prosodic-related information from the variable-length input. The latent query vectors are equivalent to a fixed-capacity representation space within the encoder used to hold prosodic information; their number is fixed while the input length is variable, thus structurally providing the foundation for "mapping variable-length input to fixed-length representation."

[0046] S143. Cross-attention processing is performed on the latent query vector and variable-length bimodal input data to extract prosodic representation information from the variable-length bimodal input data.

[0047] Step S143 utilizes a cross-attention mechanism to establish an information interaction relationship between a fixed number of potential query vectors and variable-length bimodal input data, and extracts high-level representations useful for prosodic prediction from the bimodal input. The significance of cross-attention processing lies in the fact that the potential query vectors do not passively receive all input, but rather selectively aggregate information relevant to the current prosodic modeling task from acoustic features and phoneme sequences based on attention allocation relationships. Since the input simultaneously contains acoustic and textual modalities, cross-attention processing can not only capture prosodic cues such as pitch, energy, and rhythm in the reference audio, but also combine phoneme structure, speech unit boundaries, and textual conditional information in the reference text to jointly complete representation extraction. Thus, the information output by the encoder no longer remains at the local frame-level features or simple text symbol-level features, but gradually forms joint representation information oriented towards the prosodic modeling task, laying the foundation for subsequent compression and generation of potential prosodic embeddings.

[0048] S144. The extracted prosodic representation information is compressed to generate a fixed-length latent prosodic embedding.

[0049] In step S144, the prosodic representation information obtained after cross-attention processing is further compressed into a fixed-length latent prosodic embedding to form a high-level representation that can represent the overall prosodic style and pattern of the reference speech. Since the prosodic representation information extracted in the previous stage still originates from variable-length bimodal input, it contains local information that varies with the input length. Therefore, compression processing is needed to integrate the scattered local prosodic representations into a unified-dimensional latent representation. The latent prosodic embedding can be understood as an abstract generalization of the prosodic patterns in the reference audio and text. It no longer directly depends on the original input length but retains the style and prosodic information most relevant to the subsequent prediction task in a fixed-length form. Through this process, the model can map the input samples to a unified representation space while maintaining its prosodic semantic expressiveness, thereby improving the processing efficiency and representation stability of the subsequent decoder when performing mask recovery, prosodic transfer, and target text prosodic prediction.

[0050] As an example, in a fintech scenario, reference speech and its corresponding text from financial digital human customer service, intelligent voice outbound calls, or risk disclosure broadcasts can be selected as input samples. The acoustic feature sequence after synchronous masking and the masked phoneme sequence together constitute variable-length bimodal input data, which is then fed into a prosodic encoder. Since the duration, speech rate, and text length of different customer service statements typically vary, the sequence length of the input data is not fixed. Therefore, a fixed number of latent query vectors are pre-set within the prosodic encoder to actively aggregate prosodic style-related information from bimodal inputs of different lengths. During encoding, the latent query vectors can perform cross-attention processing on information such as pitch fluctuations, energy changes, and rhythm distribution in the acoustic features, as well as the boundaries of articulatory units and textual expression structures in the phoneme sequence, thereby extracting prosodic representation information that can collectively characterize the tone of financial services. Subsequently, the extracted prosodic representation information is compressed to generate a fixed-length latent prosodic embedding, which serves as a high-level representation for subsequent prosodic recovery and target text prosodic prediction. Similarly, in digital healthcare scenarios, reference voices and their corresponding texts from patient guidance broadcasts, health education, medication reminders, or rehabilitation guidance can be selected as input samples to form variable-length bimodal input data in the same way. Then, a fixed number of latent query vectors are used to extract prosodic representation information related to the stable, friendly, and suggestive medical voice style from the bimodal input, and then compressed to generate a fixed-length latent prosodic embedding.

[0051] Through steps S141 to S144, acoustic and textual information of varying lengths and heterogeneous structures can be mapped into a unified-dimensional latent prosodic representation, thus establishing a stable representation foundation for prosodic modeling tasks within the model. Since the acoustic feature sequence corresponding to the reference audio differs from the phoneme sequence corresponding to the reference text in length, granularity, and representation, directly using the original input for subsequent predictions can easily cause the model's computational complexity to fluctuate with the input length and increase inconsistencies in the representation space between different samples. By pre-introducing a fixed number of latent query vectors and using cross-attention processing to extract prosodic-related information from the variable-length bimodal input, the encoding process no longer depends on the original length of the input sequence but instead aggregates information around a fixed-capacity latent representation unit, thereby achieving the conversion from variable-length input to fixed-length representation. Furthermore, the pitch, energy, and duration variation information in the acoustic modality and the phoneme structure information in the text modality are jointly absorbed and compressed in the same encoding process. The resulting latent prosodic embedding not only preserves the overall prosodic style of the reference speech but also improves the coupling degree and expressive compactness of cross-modal information. The subsequent decoding stage can perform prosodic recovery and target text prediction in a unified potential representation space, reducing processing instability caused by changes in input length and improving the model's adaptability to style transfer, prosodic abstraction, and zero-shot prediction tasks. Ultimately, this helps to improve the processing efficiency, representation stability, and generation naturalness of prosodic prediction.

[0052] S15. Embed the latent prosody into the input of the first prosody decoder and the second prosody decoder, and predict the prosodic features corresponding to the masked part based on the masked prosodic input and the masked text, respectively.

[0053] As an alternative implementation method, please continue reading. Figure 3 , Figure 3 This is a schematic diagram of the execution flow of predicting prosodic features in the speech prosodic prediction model training method provided in the embodiments of this application. The process of predicting prosodic features in step S15 above may also include the following steps S151 to S154.

[0054] S151. Input the latent prosodic embedding and masked prosodic input into the first prosodic decoder to predict the first prosodic feature corresponding to the masked portion.

[0055] Step S151 establishes a prediction path centered on acoustic prosodic recovery, enabling the first prosodic decoder to reconstruct the prosodic features corresponding to the masked region based on latent prosodic embeddings and masked prosodic inputs. The latent prosodic embeddings originate from the pre-encoding stage and carry the overall prosodic style, rhythmic pattern, and high-level representation information of the reference speech; the masked prosodic inputs retain the local prosodic context of the unmasked portion. By simultaneously utilizing high-level latent representations and local residual prosodic information, the first prosodic decoder can perform directional recovery around the missing region, thereby outputting the first prosodic features corresponding to the masked portion. The significance of step S151 lies in enabling the model to not only rely on textual information for prosodic prediction but also learn the recovery rules of missing segments from the existing prosodic patterns of the reference speech, thereby enhancing the latent prosodic embeddings output by the encoder's capabilities in carrying acoustic prosodic information.

[0056] S152. Calculate the first training loss based on the first prosodic feature and the target prosodic feature.

[0057] In step S152, the predicted result output by the first prosodic decoder is compared with the corresponding target prosodic features in the reference audio to establish an error metric for the first prediction path, thereby providing an optimization basis for subsequent parameter updates. The first prosodic feature reflects the model's recovery result of the masked prosodic information in the current training round, while the target prosodic feature reflects the true prosodic pattern of the reference audio at the same position. The difference between the two can directly characterize the degree of fit of the current prediction path on the prosodic recovery task. By calculating the first training loss, the question of "whether the first prosodic decoder effectively recovers the missing prosodic information" can be transformed into a quantifiable training signal, so that subsequent model updates no longer remain at the structural level of connection relationships, but are based on prediction error feedback.

[0058] S153. Update the prosodic encoder based on the first training loss.

[0059] In step S153, the prosodic encoder is optimized using the first training loss to make the latent prosodic embedding output by the encoder more suitable for the acoustic prosodic recovery task. Since the first training loss directly originates from the prediction result of the first prosodic decoder on the prosodic features of the masked portion, this loss can inversely characterize whether the latent prosodic embedding is sufficient in carrying acoustic prosodic information. Using the first training loss to update the prosodic encoder means that the optimization focus is not only on the decoding end, but also further applied to the latent representation learning process at the encoding end, enabling the prosodic encoder to output a higher-level representation that is more suitable for recovering missing prosodic information in subsequent training. Through this update mechanism, the latent prosodic embedding is no longer merely a compressed result of the input information, but gradually evolves into an intermediate representation that combines overall style generalization ability and local prosodic recovery support ability, improving the training effect and representation stability of subsequent prediction paths.

[0060] S154. Input the latent prosodic embedding and masked text into the second prosodic decoder to predict the second prosodic features corresponding to the masked portion.

[0061] In step S154, a text-condition-driven prediction path is established, enabling the second prosodic decoder to predict prosodic features corresponding to the masked portion based on latent prosodic embeddings and masked text. Unlike the first prediction path, which focuses on recovering missing information from existing prosodic input, the second prediction path emphasizes utilizing the phonetic structure, phoneme boundaries, and textual conditional information within the text content, combined with the overall style representation contained in the latent prosodic embedding, to infer the target prosodic features. Furthermore, the second prosodic decoder not only undertakes the mask recovery task but also plays a role in coupling and modeling textual conditions with high-level latent prosodic representations. Through this prediction path, the model can learn the mapping relationship between textual information and prosodic patterns, enabling it to generate corresponding prosodic features based on latent prosodic embeddings and textual conditions even in the absence of complete reference prosodic input when facing target text during the inference stage. This provides support for the practical application of speech prosodic prediction methods.

[0062] As an example, during the training phase, reference voices can be selected from financial customer service announcements, risk warnings, and product explanations in fintech scenarios, or from triage announcements, health education messages, and medication reminders in digital healthcare scenarios. The corresponding scripts, educational texts, or prompt texts are then input into the model as reference texts. After a pre-encoding process, the model first generates a latent prosodic embedding. This latent prosodic embedding, along with the masked prosodic input, is then fed into the first prosodic decoder to recover and predict the prosodic features corresponding to the masked positions. For example, in financial risk warning statements, the model can recover missing prosodic information based on pauses, emphasis, and rhythmic changes; in medical reminder statements, the model can recover corresponding prosodic information based on a steady tone, suggestive stress, and the degree of relaxation in the speech. Subsequently, the first prosodic feature output by the first prosodic decoder is compared with the target prosodic feature corresponding to the reference voice at the same position. A first training loss is calculated, and this loss is used to update the prosodic encoder, enabling the encoder to gradually learn the latent prosodic representation suitable for financial announcements or medical reminders. Building upon this foundation, the latent prosodic embedding and the masked text are then input into a second prosodic decoder. This second decoder, working in conjunction with the overall prosodic style representation and textual conditional information, predicts the prosodic features corresponding to the masked locations. Thus, the model can learn both how to recover missing prosodic information from reference speech and how to infer corresponding prosodic patterns from specific business or medical texts, providing a training foundation for subsequent speech prosodic prediction of target texts.

[0063] Through steps S151 to S154, an optimization path for acoustic prosodic recovery and an optimization path for text-conditional prediction can be established simultaneously during training, enabling the model to learn a more stable and transferable prosodic representation. Specifically, the first prediction path, based on latent prosodic embedding and masked prosodic input, recovers missing regions and directly applies the prosodic encoder through error feedback with the target prosodic features. This allows the latent representation output by the prosodic encoder to gradually acquire the ability to effectively carry the overall style, local rhythmic variations, and prosodic patterns of the reference speech, rather than simply being a compressed result of the input information. The second prediction path further introduces text conditions on the basis of latent prosodic embedding, enabling the model to learn the correspondence between text structure, articulatory units, and prosodic patterns. Since the two paths constrain the same prosodic modeling task from the perspectives of acoustic context recovery and text-conditional inference, the model can reduce its dependence on a single information source during training and enhance the comprehensive expressive ability of the latent prosodic representation to style information and text prosodic cues. In the subsequent inference stage, when faced with target text, the model can still generate relatively accurate, stable and natural prosodic features by relying on the latent prosodic representation and text mapping relationship formed during the training stage, even in the absence of complete reference prosodic input, thereby improving the prosodic prediction effect under zero-sample conditions.

[0064] As another optional implementation, the process of predicting the second prosodic feature in step S154 above may also include the following steps S1541 to S1544.

[0065] S1541. Set the conditional transformer in the second prosody decoder.

[0066] In step S1541, a condition-oriented feature modulation structure is introduced into the second prosodic decoder. This allows the second prosodic decoder to not only perform feature transformation using the conventional decoding process when performing prosodic prediction under text conditions, but also to dynamically adjust the internal feature representation based on the latent prosodic embedding. Since the second prosodic decoder is responsible for predicting the prosodic features corresponding to the mask portion based on text conditions and latent prosodic representation, it needs not only sequence modeling capabilities but also the ability to inject, control, and adjust style and prosodic conditions. By setting a conditional transformer in the second prosodic decoder, high-level prosodic style information carried in the latent prosodic embedding can be introduced into the feature transformation process. This allows text-condition-driven prosodic prediction to no longer rely solely on the text itself but to be simultaneously constrained by the overall prosodic style, thus providing a structural basis for subsequent fine-grained time-varying modulation.

[0067] S1542. Set up a cross-attention module in the conditional transformer, and use the latent prosodic embedding as the query vector and the input features of the second prosodic decoder as the key and value.

[0068] In step S1542, a targeted information association is established between the latent prosodic embedding and the input features of the second prosodic decoder. This enables the conditional transformer to extract prosodic control-related content from the input features, centered around the current prosodic modeling task. The latent prosodic embedding carries the overall style, rhythmic pattern, and high-level prosodic information of the reference speech. Therefore, as a query vector, it allows subsequent attention calculations to focus on the goal of "what prosodic style needs to be injected." The input features of the second prosodic decoder reflect the local contextual information of the position to be predicted under the current text conditions. As keys and values, they help establish a correlation between the text content and the overall prosodic style. By setting a cross-attention module in the conditional transformer, the second prosodic decoder can no longer simply perform static conditional injection but can selectively extract information and conditionally fuse the input features around the latent prosodic embedding, thus laying the foundation for subsequent generation of modulation parameters frame by frame.

[0069] S1543. Based on the query vector, key and value, predict the scaling and translation parameters corresponding to different time frames through the cross-attention module.

[0070] In step S1543, the conditional transformer (e.g., a Transformer) generates dynamic modulation parameters oriented towards the time dimension, thereby overcoming the limitation of relying solely on global conditional control. Since prosodic variations typically manifest as local patterns that change over time—for example, the intensity of accents, the magnitude of energy changes, and pitch trends may differ at different locations—generating a uniform set of parameters based on a single global condition and applying it to all time frames often fails to accurately represent fine-grained prosodic dynamics. By performing cross-attention processing based on query vectors, keys, and values, the conditional transformer can calculate modulation information related to the current frame at different time positions, thereby predicting independent scaling and translation parameters. Thus, the modulation parameters obtained by the model are no longer a uniform control result for the entire input segment, but are transformed into dynamic control signals per time frame and per local segment, which helps improve the temporal resolution of prosodic modeling.

[0071] S1544. Based on the scaling and translation parameters corresponding to different time frames, perform time-varying modulation on the features in the conditional transformer to predict the second prosodic features corresponding to the mask part.

[0072] In step S1544, the time-frame modulation parameters generated in the previous steps are applied to the feature representation in the conditional transformer, enabling the second prosodic decoder to achieve variable and distinguishable prosodic control capabilities in the time dimension. Scaling parameters adjust the amplitude of feature responses at different time positions, while translation parameters adjust the degree of offset in feature distribution at different time positions. When these two parameters work together on the features in the conditional transformer, the feature representation can vary differentially with different time positions. Through this process, the second prosodic decoder can achieve finer-grained control over local prosodic patterns in different time frames under the constraint of the overall latent prosodic style, thereby generating second prosodic features corresponding to the masked portion. In the text-conditionally driven prediction path, the model not only retains the overall style information provided by the latent prosodic embedding but also further reflects the local prosodic variation patterns at different time positions, thus improving the accuracy and naturalness of the second prosodic features.

[0073] As an example, in fintech scenarios, reference speech and its corresponding text from financial digital human customer service, intelligent voice outbound calls, or risk warning broadcasts can be used as training samples. A conditional transformer is introduced into the second prosodic decoder, enabling the model to not only rely on the literal information of the text itself for prosodic prediction when processing text content such as "risk disclosure," "explanation of profit fluctuations," and "business confirmation," but also to conditionally control the decoding features by incorporating the broadcasting style, emphasis positions, and rhythmic patterns reflected in the reference speech. Specifically, latent prosodic embeddings can be used as high-level conditional information reflecting the overall tone style and prosodic pattern, and a cross-attention relationship can be established with the input features in the second prosodic decoder. This allows the model to generate corresponding scaling and translation parameters around different time positions, and then modulate the features in the conditional transformer frame-by-frame based on these parameters. This results in more pronounced stress changes in "risk warning" text at emphasized positions, and a smoother tone and rhythm in "product explanation" text during continuous narration. Similarly, in digital healthcare scenarios, reference speech and its corresponding text from patient guidance announcements, health education, medication reminders, or rehabilitation guidance can be used as training samples. This allows the conditional transformer to differentially modulate the decoding features at different time frames based on the smooth, friendly, or suggestive speech style carried by the latent prosodic embedding. For example, in medical prompts such as "take medication on time," "follow-up appointment schedule," or "examination instructions," the model can create clearer emphasis at key prompt positions and maintain a smoother intonation variation at continuous explanatory positions.

[0074] Through steps S1541 to S1544, a fine-grained modulation link from the overall prosodic style to local temporal frame features can be established within the second prosodic decoder. This allows the latent prosodic embedding to no longer participate in prediction merely as a static condition, but to further influence the feature transformation process at different temporal positions. Since prosodic variations under textual conditions typically exhibit significant temporal differences, with inconsistencies in stress intensity, energy distribution, intonation fluctuations, and rhythmic beats across different positions, relying solely on a single global condition to apply uniform control to all temporal frames can easily lead to insufficient expression of local prosodic variations, thus limiting the prediction results' ability to approximate the dynamic patterns of real speech. After generating time-frame-varying scaling and translation parameters based on cross-attention relationships, the conditional transformer can form differentiated feature modulations for different positions, enabling a tighter coupling between overall style information and local textual context in the temporal dimension. Therefore, the second prosodic features output by the second prosodic decoder can not only maintain the consistency of the overall prosodic style of the reference speech, but also reflect the local prosodic details that change over time. This improves the temporal resolution, feature expression accuracy and generation naturalness of prosodic prediction under text conditions, providing a more stable and detailed feature basis for subsequent speech prosodic prediction of target text.

[0075] S16. Determine the target training loss based on the predicted prosodic features and the target prosodic features, and update the parameters of the speech prosodic prediction model according to the target training loss to obtain the trained speech prosodic prediction model.

[0076] As an optional implementation, the step of updating the parameters of the speech prosody prediction model in step S16 above may also include the following steps S161 to S164.

[0077] S161. Based on the speech prosody prediction model, train the first prosodic feature and the speech prosody prediction model training target prosodic feature, and calculate the first training loss corresponding to the training mask part of the speech prosody prediction model.

[0078] In step S161, an error metric mechanism is established for the prediction path corresponding to the first prosodic decoder to quantify the degree to which the first prosodic feature recovers the prosodic information of the masked region. The first prosodic feature originates from the prediction result driven by the latent prosodic embedding and the masked prosodic input, reflecting the model's ability to reconstruct missing regions under the existing prosodic context of the reference speech. The target prosodic feature corresponds to the true prosodic information of the reference audio at the same position and can serve as a supervision benchmark during the training phase. By calculating the difference between the first prosodic feature and the target prosodic feature, the fitting effect of the first prediction path in the masked recovery task can be transformed into a loss signal that can be used for optimization. This loss signal not only reflects the current model's accuracy in recovering the local prosodic structure but also provides a basis for subsequent targeted updates to the prosodic encoder, thereby enabling the latent prosodic embedding to gradually acquire stronger acoustic prosodic representation capabilities.

[0079] S162. Based on the second prosodic feature and the target prosodic feature of the speech prosodic prediction model, train the second prosodic feature and calculate the second training loss corresponding to the training mask part of the speech prosodic prediction model.

[0080] In step S162, an error feedback mechanism is established for the prediction path corresponding to the second prosodic decoder to quantify the difference between the prosodic prediction result driven by text conditions and the true target. The second prosodic feature is the prediction result obtained under the combined effect of latent prosodic embedding and masked text, focusing on reflecting the model's ability to infer missing regions using text conditions and overall prosodic style; the target prosodic feature serves as the true supervision information at the corresponding position, used to measure the accuracy of the prediction path. By calculating the second training loss, the text-side conditional modeling capability can be directly incorporated into the training optimization process, enabling the model to not only learn how to preserve the overall style features in the reference speech, but also how to map the articulation structure, intonation boundaries, and prosodic cues in the text content to the corresponding prosodic output. Therefore, the second prediction path and the first prediction path form error feedback for different information sources during training, which helps to improve the model's ability to comprehensively utilize bimodal prosodic information.

[0081] S163. Calculate the training target loss of the speech prosody prediction model based on the first training loss of the speech prosody prediction model and the second training loss of the speech prosody prediction model.

[0082] In step S163, the error feedback from the two prediction paths is further summarized into a unified training objective for overall model optimization. The first training loss primarily reflects the model's ability to recover missing information based on acoustic prosodic context, while the second training loss primarily reflects the model's ability to infer prosodic features based on textual conditions. These two losses correspond to different modeling focuses within the speech prosodic prediction model. Training based solely on one type of loss can easily lead to a bias towards a single information source, thus weakening the constraining effect of the other prediction path in the overall training process. By incorporating both the first and second training losses into the calculation of the target training loss, the speech prosodic prediction model can simultaneously consider both acoustic recovery capabilities and textual conditional prediction capabilities during parameter optimization, ensuring the training objective covers the main optimization directions involved in dual-modal joint modeling. Therefore, the target training loss can serve as a comprehensive error metric for the entire training phase, uniformly guiding subsequent parameter updates.

[0083] S164. Based on the training loss of the speech prosody prediction model training objective, update the parameters of the speech prosody prediction model training prosody encoder, the speech prosody prediction model training first prosody decoder, and the speech prosody prediction model training second prosody decoder to obtain the trained speech prosody prediction model.

[0084] In step S164, the overall parameters of the speech prosody prediction model are optimized using the target training loss. This allows the prosody encoder, the first prosody decoder, and the second prosody decoder to gradually form a collaborative working relationship suitable for zero-shot prosody prediction tasks during joint training. The prosody encoder is responsible for extracting latent prosodic embeddings from the variable-length bimodal input; the first prosody decoder is responsible for recovering missing regions based on acoustic prosodic context; and the second prosody decoder is responsible for predicting corresponding prosodic features by combining textual conditions and latent prosodic representations. Therefore, although the three components perform different functions in the model, their parameters need to be coordinated through a unified optimization objective during training. After updating the parameters of the three components based on the target training loss, the speech prosody prediction model can gradually reduce the difference between the prediction results and the true target, and improve the consistency between latent representation learning, local prosodic recovery, and textual conditional prediction. After continuous iterative training, a trained speech prosody prediction model can be obtained, providing a model foundation for performing prosody prediction on target text in the subsequent inference stage.

[0085] As an example, in fintech scenarios, financial avatar announcements, risk warnings, or product explanations can be selected as reference audio, with the corresponding text input into the model. In digital healthcare scenarios, triage announcements, health education messages, medication reminders, or rehabilitation guidance can be selected as reference audio, with the corresponding medical text input into the model. After pre-encoding and dual-path prediction, the model outputs first prosodic features based on latent prosodic embedding and masked prosodic input, and second prosodic features based on latent prosodic embedding and masked text output. Subsequently, the first prosodic features are compared with the true prosodic information of the reference audio at the corresponding masked position to obtain the first training loss reflecting the acoustic recovery path prediction error; simultaneously, the second prosodic features are compared with the true prosodic information at the same position to obtain the second training loss reflecting the text conditional prediction path error. Based on this, the two types of training losses are combined to form the target training loss, which is then used to jointly update the parameters of the prosodic encoder, the first prosodic decoder, and the second prosodic decoder. Through the above training process, the model can learn the common emphasis, pauses and rhythm changes in financial broadcasting tone, as well as the common smooth, friendly and suggestive prosodic patterns in medical prompts, thereby gradually forming a speech prosodic prediction capability applicable to different business texts and different application scenarios.

[0086] Through steps S161 to S164, the error feedback generated by the two prediction paths can be uniformly incorporated into the same training optimization framework. This allows the speech prosody prediction model to simultaneously consider both acoustic prosody recovery capability and text-conditionally driven prosody prediction capability during parameter updates. Specifically, the first training loss mainly reflects the error when the model recovers the missing region based on the existing prosodic context, while the second training loss mainly reflects the error when the model infers the prosodic features of the missing region based on text conditions. These two losses correspond to modeling capabilities under different information sources. After further summarizing the two types of training losses into a target training loss, the model optimization objective is no longer limited to the local fitting effect of a single path, but can cover the joint optimization needs of overall style representation learning, local prosodic recovery, and text conditional mapping. Therefore, the prosodic encoder, the first prosodic decoder, and the second prosodic decoder can form a more coordinated parameter relationship during continuous training, enabling the latent prosodic embedding to both carry the overall prosodic style in the reference speech and support the prosodic prediction of the target text in the subsequent decoding stage. Based on this training mechanism, the model's dependence on single-modal information can be reduced, the stability and consistency of dual-modal joint modeling can be improved, and the accuracy of prosody prediction, style transfer ability and generation naturalness of the trained model under zero-sample conditions can be enhanced.

[0087] The speech prosody prediction model training method provided in this application constructs an encoder-decoder prosody prediction architecture independent of a specific speech synthesis system. It incorporates acoustic information from reference audio and phoneme structure information from reference text into the same training framework and utilizes a mask prediction mechanism to drive bimodal joint representation learning. This enables the model to extract potential representations from reference speech that contain both overall style and local prosodic patterns. Compared to existing solutions that rely solely on reference audio for style extraction or are strongly coupled with a specific text-to-speech system, this solution, on the one hand, uses a synchronous masking mechanism based on phoneme alignment to ensure consistency between missing articulatory units in the text modality and corresponding prosodic segments in the acoustic modality, thereby reducing training noise caused by cross-modal misalignment. On the other hand, it compresses bimodal inputs of varying lengths and structures into fixed-length potential prosodic embeddings through a fixed number of potential query vectors and cross-attention processing. This forms a stable high-level prosodic representation unaffected by changes in input length, alleviating the high complexity of processing variable-length sequences and improving the ability to abstract and transfer style information. Furthermore, this scheme sets up two prediction paths: a first prosodic decoder and a second prosodic decoder. This allows the model to simultaneously accept acoustic prosodic recovery constraints and textual conditional prediction constraints during training. The first path uses masked prosodic input to recover missing regions and feeds back the corresponding errors to the encoder, enhancing the latent prosodic embedding's acoustic prosodic capacity. The second path combines masked text for prediction, improving the model's ability to learn the mapping relationship between text structure, articulatory units, and prosodic patterns. Simultaneously, a conditional transformer is introduced within the second prosodic decoder, and a cross-attention mechanism is used to generate independent scaling and translation parameters for different time frames. This allows the overall style conditions to be further transformed into fine-grained dynamic modulation in the temporal dimension, thereby improving the accuracy of expressing local prosodic variations. Based on the above training mechanism, the model no longer obtains a single-source, single-level prosodic representation, but a potential prosodic representation that combines bimodal collaboration, overall style abstraction, and local time-varying control capabilities. Therefore, in the subsequent inference stage, when faced with no speaker, no text content, or text from different scenarios, it can generate more stable, natural, and style-consistent prosodic features without retraining or fine-tuning. This simultaneously improves the model's decoupling, zero-shot generalization ability, naturalness of prosodic prediction, and robustness in complex scenarios.

[0088] Please continue reading. Figure 4 , Figure 4 This is a schematic diagram of the execution flow of the speech prosody prediction method provided in the embodiments of this application, as shown below. Figure 4 As shown, the process includes steps S21 to S25.

[0089] S21. Obtain the reference audio, reference text, and target text.

[0090] S22. Extract the acoustic features of the training reference audio of the speech prosody prediction model, convert the training reference text of the speech prosody prediction model into a reference phoneme sequence, and align the training acoustic features of the speech prosody prediction model with the training reference phoneme sequence.

[0091] In this process, steps S21 and S22 establish a data input foundation consistent with that of the training phase during the inference phase. Reference audio and reference text are used together to characterize the overall prosodic style, rhythmic pattern, and textual structure information of the reference speech, while the target text provides textual conditions for the content to be predicted. By extracting acoustic features from the reference audio and converting the reference text into a reference phoneme sequence before alignment, a correspondence can be established between the acoustic changes in the reference speech and the articulatory units in the text, thus ensuring that the data format input to the trained model remains consistent with that of the training phase. This reduces representational bias caused by inconsistencies between the training and inference distributions and provides a complete and stable bimodal input foundation for subsequent extraction of potential prosodic representations.

[0092] S23. Input the training acoustic features of the speech prosody prediction model and the training reference phoneme sequence of the speech prosody prediction model into the prosody encoder in the trained speech prosody prediction model to obtain a fixed-length latent prosody embedding.

[0093] In step S23, a transferable high-level prosodic representation is extracted from the reference speech during the inference phase. Since the acoustic feature sequence corresponding to the reference audio differs in length and representation from the phoneme sequence corresponding to the reference text, the trained prosodic encoder, through joint encoding of the bimodal input, maps the originally variable-length, heterogeneous input to a fixed-length latent prosodic embedding. This latent prosodic embedding is essentially an abstract generalization of the overall style, local rhythmic patterns, and prosodic variation patterns of the reference speech; therefore, it serves as a style carrier and prosodic condition during the inference phase. Based on this processing, the model can stably transfer the prosodic style from the reference speech to the prediction process of the subsequent target text without relying on the original input length, thereby improving style preservation ability and inference stability under zero-shot conditions.

[0094] S24. Input the target text of the speech prosody prediction model training and the latent prosody of the speech prosody prediction model training into the prosody decoder of the trained speech prosody prediction model to obtain the prosodic features corresponding to the target text of the speech prosody prediction model training.

[0095] S25. Output the prosodic features trained by the speech prosodic prediction model as the speech prosodic prediction result corresponding to the target text of the speech prosodic prediction model. The trained speech prosodic prediction model is obtained through the above-described speech prosodic prediction model training method.

[0096] In this process, steps S24 and S25 establish a direct coupling relationship between the target text content and the reference speech style during the inference stage. The latent prosodic embedding provides overall prosodic style information from the reference speech, while the target text provides the textual conditions for the content to be generated. The prosodic decoder predicts the prosodic features corresponding to the target text based on the combined effect of both. Therefore, the prediction result no longer depends solely on the target text itself, but can simultaneously inherit the intonation trend, rhythm distribution, and style features from the reference speech, achieving a balance between content adaptability and style consistency in the output. Furthermore, outputting the predicted prosodic features as the speech prosodic prediction result corresponding to the target text can directly provide a basis for prosodic control in subsequent speech generation, speech playback, or digital human speech interaction modules. This reduces the problems of monotonous tone, rigid rhythm, or inconsistent style caused by purely text-driven prediction, and improves naturalness, stability, and usability in practical applications.

[0097] As an example, in a fintech scenario, a speech prosody prediction model can be trained using historical audio recordings of a financial avatar customer service representative, corresponding text as reference text, and new risk warning texts, product explanation texts, or business confirmation texts as target text inputs. The model first extracts latent prosodic embeddings representing the customer service representative's speaking style from the reference audio and reference text, then combines these with the target text to generate prosodic features that match the target text. This ensures that the prediction results retain the pauses, emphasis, and tone of the reference audio while adapting to the specific content of the target text. Similarly, in a digital healthcare scenario, triage announcements, health education messages, medication reminders, or rehabilitation guidance messages can be used as reference audio, corresponding text as reference text, and the consultation prompts, follow-up visit reminders, or examination instructions to be broadcast as target text inputs to the model. By encoding reference speech styles, the model can extract stable, friendly, or suggestive latent prosodic representations. These representations are then combined with the target text to generate corresponding prosodic features. This ensures that the output maintains overall stylistic consistency in medical speech while reflecting local intonation, stress, and rhythm variations adapted to the specific text content. Through this approach, the trained speech prosodic prediction model can still achieve style transfer and prosodic prediction based on a small number of reference speeches when faced with new texts with different content. This provides consistent and highly natural prosodic prediction results for applications such as financial digital human broadcasting, intelligent customer service voice interaction, medical triage broadcasting, and health education voice generation.

[0098] The speech prosody prediction method provided in this application combines the prosodic style information provided by the reference audio and reference text with the content information provided by the target text. This enables the trained speech prosody prediction model to generate prosodic features consistent with the reference speech style for the target text under zero-sample conditions. Since the model called during the inference stage has undergone bimodal joint training and bipath prediction constraint optimization, it can not only maintain the overall prosodic style of the reference speech but also adapt to the local intonation, rhythm, and stress changes corresponding to the target text, thereby improving the naturalness, stability, and style consistency of the speech prosody prediction results.

[0099] Please continue reading. Figure 5 , Figure 5 This is a schematic diagram of the system structure of the speech prosody prediction model training device provided in the embodiments of this application, as shown below. Figure 5 As shown, the speech prosody prediction model training device 50 includes: a model building module 51, a data acquisition and processing module 52, a synchronization mask module 53, an encoding and compression module 54, a dual decoding prediction module 55, and a loss calculation and update module 56.

[0100] The model building module 51 is specifically used to build a speech prosodic prediction model, which includes a prosodic encoder, a first prosodic decoder, and a second prosodic decoder. The data acquisition and processing module 52 is specifically used to acquire reference audio and reference text, extract the acoustic features of the reference audio and the target prosodic features corresponding to the reference audio, convert the reference text into a reference phoneme sequence, and align the acoustic features and the reference phoneme sequence. The synchronization masking module is specifically used to perform synchronization masking processing on the features to be masked in the acoustic features and the reference phoneme sequence based on the alignment relationship between the acoustic features and the reference phoneme sequence, to obtain a mask. The system comprises: acoustic features and a masked reference phoneme sequence; the encoding and compression module specifically inputs the masked acoustic features and the masked reference phoneme sequence into the prosody encoder to encode and compress the variable-length bimodal input to obtain a fixed-length latent prosodic embedding; the dual-decoding prediction module specifically inputs the latent prosodic embedding into the first prosodic decoder and the second prosodic decoder to predict the prosodic features corresponding to the masked portion based on the masked prosodic input and the masked text, respectively; and the loss calculation and update module specifically determines the target training loss based on the predicted prosodic features and the target prosodic features, and updates the parameters of the speech prosodic prediction model according to the target training loss to obtain the trained speech prosodic prediction model.

[0101] As an optional implementation, the synchronization masking module 53 is further specifically used to determine the reference phoneme to be masked according to the phoneme dimension; based on the alignment relationship, determine the acoustic features aligned with the reference phoneme to be masked as the features to be masked; and perform synchronization masking processing on the reference phoneme to be masked and the features to be masked so that the acoustic features corresponding to a single reference phoneme are either all masked or all unmasked.

[0102] As an optional implementation, the encoding compression module 54 is further specifically used to determine the mask acoustic features and the mask reference phoneme sequence as variable-length bimodal input data, and input the variable-length bimodal input data to the prosodic encoder; set a fixed number of latent query vectors in the prosodic encoder; perform cross-attention processing on the latent query vectors and the variable-length bimodal input data to extract prosodic representation information from the variable-length bimodal input data; compress the extracted prosodic representation information to generate a fixed-length latent prosodic embedding.

[0103] As an optional implementation, the dual-decoding prediction module 55 is further configured to input the latent prosodic embedding and the masked prosodic input into the first prosodic decoder to predict the first prosodic feature corresponding to the masked portion; calculate a first training loss based on the first prosodic feature and the target prosodic feature; update the prosodic encoder according to the first training loss; and input the latent prosodic embedding and the masked text into the second prosodic decoder to predict the second prosodic feature corresponding to the masked portion.

[0104] As an optional implementation, the dual-decoding prediction module 55 is further configured to set a conditional transformer in the second prosodic decoder; set a cross-attention module in the conditional transformer, and use the latent prosodic embedding as a query vector and the input features of the second prosodic decoder as keys and values; based on the query vector and the keys and values, predict the scaling and translation parameters corresponding to different time frames through the cross-attention module; and perform time-varying modulation on the features in the conditional transformer according to the scaling and translation parameters corresponding to different time frames to predict the second prosodic features corresponding to the mask portion.

[0105] As an optional implementation, the loss calculation and update module 56 is further specifically used to calculate a first training loss corresponding to the masked portion based on the first prosodic feature and the target prosodic feature; calculate a second training loss corresponding to the masked portion based on the second prosodic feature and the target prosodic feature; calculate the target training loss according to the first training loss and the second training loss; and update the parameters of the prosodic encoder, the first prosodic decoder, and the second prosodic decoder according to the target training loss to obtain the trained speech prosodic prediction model.

[0106] As another optional implementation, this application embodiment also provides a speech prosody prediction device, including: a data acquisition module for acquiring reference audio, reference text, and target text; a feature extraction and alignment module for extracting acoustic features of the reference audio, converting the reference text into a reference phoneme sequence, and aligning the acoustic features and the reference phoneme sequence; a prosody encoding module for inputting the acoustic features and the reference phoneme sequence into a prosody encoder in a trained speech prosody prediction model to obtain a fixed-length latent prosody embedding; a prosody decoding module for inputting the target text and the latent prosody embedding into a prosody decoder in the trained speech prosody prediction model to obtain prosodic features corresponding to the target text; and a result output module for outputting the prosodic features as a speech prosody prediction result corresponding to the target text.

[0107] It should be noted that the above-mentioned speech prosody prediction model training device can execute the speech prosody prediction model training method provided in the embodiments of this application, and the speech prosody prediction device can execute the speech prosody prediction method provided in the embodiments of this application, possessing the corresponding functional modules and beneficial effects of the execution method. Technical details not described in detail in the embodiments of the speech prosody prediction model training device and the speech prosody prediction device can be found in the speech prosody prediction model training method and the speech prosody prediction method provided in the embodiments of this application.

[0108] Figure 6 This is a schematic diagram of the hardware structure of an electronic device for performing a speech prosody prediction model training method or a speech prosody prediction method, as provided in the embodiments of this application. Figure 6 As shown, the electronic device 600 includes: One or more processors 610 and memory 620, Figure 6 Take the 610 processor as an example.

[0109] The processor 610 and the memory 620 can be connected via a bus or other means. Figure 6 Taking the example of a connection between China and Israel via a bus.

[0110] The memory 620, as a non-volatile computer-readable storage medium, can be used to store non-volatile software programs, non-volatile computer-executable programs, and modules, such as the program instructions / modules corresponding to the speech prosody prediction model training method or speech prosody prediction method in the embodiments of this application. The processor 610 executes various server functions and data processing by running the non-volatile software programs, instructions, and modules stored in the memory 620, thereby implementing the speech prosody prediction model training method or speech prosody prediction method described in the above method embodiments.

[0111] The memory 620 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created based on the use of the speech prosody prediction model training device or the speech prosody prediction device. Furthermore, the memory 620 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 620 may optionally include memory remotely located relative to the processor 610, which can be connected to the speech prosody prediction model training device or the speech prosody prediction device via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0112] The one or more modules are stored in the memory 620. When executed by the one or more processors 610, they perform the speech prosody prediction model training method or speech prosody prediction method in any of the above method embodiments. For example, they perform the methods described above. Figure 2 Method steps S11 to S16, Figure 3 Method steps S151 to S154, Figure 4 Steps S21 to S25 in the method are implemented. Figure 5 The functions of modules 51-56 in the document.

[0113] The above-described product can perform the methods provided in the embodiments of this application, and has the corresponding functional modules and beneficial effects for performing the methods. Technical details not described in detail in this embodiment can be found in the methods provided in the embodiments of this application.

[0114] This application provides a non-volatile computer-readable storage medium storing computer-executable instructions that are executed by one or more processors, for example... Figure 6 One of the processors 610 can enable the one or more processors to execute the speech prosody prediction model training method or speech prosody prediction method in any of the above method embodiments, for example, to execute the above-described... Figure 2 Method steps S11 to S16, Figure 3 Method steps S151 to S154, Figure 4 Steps S21 to S25 in the method are implemented. Figure 5 The functions of modules 51-56 in the document.

[0115] This application provides a computer program product, which includes a computer program stored on a non-volatile computer-readable storage medium. The computer program includes program instructions that, when executed by an electronic device, enable the electronic device to perform the speech prosody prediction model training method or speech prosody prediction method in any of the above method embodiments. For example, it can perform the above-described... Figure 2 Method steps S11 to S16, Figure 3 Method steps S151 to S154, Figure 4 Steps S21 to S25 in the method are implemented. Figure 5 The functions of modules 51-56 in the document.

[0116] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0117] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented using software and a general-purpose hardware platform, or of course, using hardware. Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0118] It should be noted that any AI models, software tools, or components not belonging to this company appearing in the embodiments of this application are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application has been authorized (with the knowledge and consent) by the relevant parties or has been fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, processing, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

[0119] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and not to limit them; under the concept of this application, the technical features of the above embodiments or different embodiments can also be combined, the steps can be implemented in any order, and there are many other variations of different aspects of this application as described above, which are not provided in detail for the sake of brevity; although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the foregoing embodiments, or make equivalent substitutions for some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.

Claims

1. A method for training a speech prosody prediction model, characterized in that, include: A speech prosody prediction model is constructed, which includes a prosody encoder, a first prosody decoder, and a second prosody decoder. Obtain reference audio and reference text, extract the acoustic features of the reference audio and the target prosodic features corresponding to the reference audio, convert the reference text into a reference phoneme sequence, and align the acoustic features and the reference phoneme sequence; Based on the alignment relationship between the acoustic features and the reference phoneme sequence, the features to be masked in the acoustic features and the reference phonemes to be masked in the reference phoneme sequence are subjected to synchronous masking processing to obtain masked acoustic features and masked reference phoneme sequences. The masked acoustic features and the masked reference phoneme sequence are input into the prosodic encoder to encode and compress the variable-length bimodal input, thereby obtaining a fixed-length latent prosodic embedding. The latent prosody is embedded into the first prosody decoder and the second prosody decoder, and the prosodic features corresponding to the masked portion are predicted based on the masked prosody input and the masked text, respectively. The target training loss is determined based on the predicted prosodic features and the target prosodic features, and the parameters of the speech prosodic prediction model are updated according to the target training loss to obtain the trained speech prosodic prediction model.

2. The method for training a speech prosody prediction model according to claim 1, characterized in that, The synchronization masking process, based on the alignment relationship between the acoustic features and the reference phoneme sequence, involves performing synchronization masking on the features to be masked in the acoustic features and the reference phonemes to be masked in the reference phoneme sequence, including: Determine the reference phonemes to be masked according to the phoneme dimension; Based on the alignment relationship, acoustic features aligned with the reference phonemes to be masked are determined as features to be masked. The reference phonemes to be masked and the features to be masked are subjected to synchronous masking processing so that the acoustic features corresponding to a single reference phoneme are either all masked or all unmasked.

3. The method for training a speech prosody prediction model according to claim 1, characterized in that, The step of inputting the masked acoustic features and the masked reference phoneme sequence into the prosodic encoder to encode and compress the variable-length bimodal input to obtain a fixed-length latent prosodic embedding includes: The mask acoustic features and the mask reference phoneme sequence are determined as variable-length bimodal input data, and the variable-length bimodal input data is input to the prosodic encoder; A fixed number of potential query vectors are set in the prosody encoder; Cross-attention processing is performed on the potential query vector and the variable-length bimodal input data to extract prosodic representation information from the variable-length bimodal input data; The extracted prosodic representation information is compressed to generate a fixed-length latent prosodic embedding.

4. The method for training a speech prosody prediction model according to claim 1, characterized in that, The step of embedding the latent prosody into the first prosody decoder and the second prosody decoder, and predicting the prosodic features corresponding to the masked portion based on the masked prosodic input and the masked text, respectively, includes: The latent prosodic embedding and the masked prosodic input are input into the first prosodic decoder to predict the first prosodic feature corresponding to the masked portion; Calculate the first training loss based on the first prosodic feature and the target prosodic feature; The prosodic encoder is updated based on the first training loss; The latent prosodic embedding and the masked text are input into the second prosodic decoder to predict the second prosodic feature corresponding to the masked portion.

5. The method for training a speech prosody prediction model according to claim 4, characterized in that, The step of inputting the latent prosodic embedding and the masked text into the second prosodic decoder to predict the second prosodic features corresponding to the masked portion includes: Set a conditional transformer in the second prosody decoder; A cross-attention module is set in the conditional transformer, and the latent prosodic embedding is used as the query vector, while the input features of the second prosodic decoder are used as the key and value. Based on the query vector, the key and the value, the scaling and translation parameters corresponding to different time frames are predicted by the cross-attention module. Based on the scaling and translation parameters corresponding to different time frames, the features in the conditional transformer are time-varyingly modulated to predict the second prosodic features corresponding to the mask portion.

6. The method for training a speech prosody prediction model according to claim 1, characterized in that, The step of determining the target training loss based on the predicted prosodic features and the target prosodic features, and updating the parameters of the speech prosodic prediction model according to the target training loss, includes: Based on the first prosodic feature and the target prosodic feature, calculate the first training loss corresponding to the mask portion; Based on the second prosodic feature and the target prosodic feature, calculate the second training loss corresponding to the mask portion; The target training loss is calculated based on the first training loss and the second training loss; Based on the target training loss, the parameters of the prosody encoder, the first prosody decoder, and the second prosody decoder are updated to obtain the trained speech prosody prediction model.

7. A method for predicting speech prosody, characterized in that, include: Obtain the reference audio, reference text, and target text; Extract the acoustic features of the reference audio, convert the reference text into a reference phoneme sequence, and align the acoustic features and the reference phoneme sequence. The acoustic features and the reference phoneme sequence are input into the prosody encoder in the trained speech prosody prediction model to obtain a fixed-length latent prosody embedding. The target text and the latent prosody are embedded into the prosody decoder of the trained speech prosody prediction model to obtain the prosody features corresponding to the target text. The prosodic features are output as the speech prosodic prediction result corresponding to the target text. The trained speech prosody prediction model is obtained by training the speech prosody prediction model training method according to any one of claims 1 to 6.

8. A training device for a speech prosody prediction model, characterized in that, include: The model building module is used to build a speech prosody prediction model, which includes a prosody encoder, a first prosody decoder and a second prosody decoder. The data acquisition and processing module is used to acquire reference audio and reference text, extract the acoustic features of the reference audio and the target prosodic features corresponding to the reference audio, convert the reference text into a reference phoneme sequence, and align the acoustic features and the reference phoneme sequence. The synchronization masking module is used to perform synchronization masking processing on the features to be masked in the acoustic features and the reference phonemes to be masked in the reference phoneme sequence based on the alignment relationship between the acoustic features and the reference phoneme sequence, so as to obtain the masked acoustic features and the masked reference phoneme sequence. The encoding and compression module is used to input the masked acoustic features and the masked reference phoneme sequence into the prosodic encoder to encode and compress the variable-length bimodal input to obtain a fixed-length latent prosodic embedding. The dual-decoding prediction module is used to embed the latent prosody into the first prosody decoder and the second prosody decoder, and predict the prosodic features corresponding to the masked portion based on the masked prosodic input and the masked text, respectively. The loss calculation and update module is used to determine the target training loss based on the predicted prosodic features and the target prosodic features, and update the parameters of the speech prosodic prediction model according to the target training loss to obtain the trained speech prosodic prediction model.

9. An electronic device, characterized in that, include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the speech prosody prediction model training method according to any one of claims 1-6, or the speech prosody prediction method according to any one of claims 7.

10. A non-volatile computer-readable storage medium, characterized in that, The non-volatile computer-readable storage medium stores computer-executable instructions, which, when executed by an electronic device, cause the electronic device to perform the speech prosody prediction model training method according to any one of claims 1-6, or the speech prosody prediction method according to any one of claims 7.

Images