Prosody prediction method, apparatus, device, storage medium and program product

Abstract

The application provides a prosody prediction method and device, equipment, storage medium and program product, and relates to artificial intelligence technology. The method comprises the following steps: performing feature extraction on a target text to obtain text features of the target text; sampling an initial prosody feature for prosody prediction from a first target distribution, and sampling noise for prosody prediction from a second target distribution; performing prosody prediction on the target text based on the text features, the initial prosody feature and the noise to obtain predicted prosody features of the target text; wherein the predicted prosody features are used for speech synthesis in combination with the text features to obtain synthesized speech of the target text with the predicted prosody features. Through the application, the diversity of the predicted prosody features of the text can be enriched, so that the synthesized speech obtained by performing speech synthesis on the text based on the predicted prosody features is diversified.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a prosody prediction method, apparatus, device, storage medium, and program product. Background Technology

[0002] Artificial Intelligence (AI) is a comprehensive technology within computer science that studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. AI technology is a multidisciplinary field encompassing a wide range of areas, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in more fields and play an increasingly important role.

[0003] Prosody prediction is also an important application of artificial intelligence. In related technologies, deterministic prosody prediction models are used to predict the prosody of text, obtaining its predicted prosodic features. However, because deterministic prosodic prediction can only predict one type of prosodic feature for a given text, the predicted prosodic features for the same text lack variation, resulting in overly simplistic synthesized speech based on these features. Summary of the Invention

[0004] This application provides a prosodic prediction method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can enrich the diversity of predicted prosodic features of text, and make the synthesized speech obtained by speech synthesis based on predicted prosodic features of text more diverse.

[0005] The technical solution of this application embodiment is implemented as follows: This application provides a prosody prediction method, including: Feature extraction is performed on the target text to obtain the text features of the target text; Initial prosodic features for prosodic prediction are sampled from the first target distribution, and noise for prosodic prediction is sampled from the second target distribution; Based on the text features, the initial prosodic features, and the noise, the prosodic features of the target text are predicted to be obtained. The predicted prosodic features are used to combine with the text features for speech synthesis to obtain synthesized speech of the target text that has the predicted prosodic features.

[0006] This application also provides a prosody prediction device, including: The feature extraction module is used to extract features from the target text to obtain the text features of the target text; The sampling module is used to sample initial prosodic features for prosodic prediction from a first target distribution and to sample noise for prosodic prediction from a second target distribution. The prosody prediction module is used to predict the prosody of the target text based on the text features, the initial prosody features, and the noise, so as to obtain the predicted prosody features of the target text. The predicted prosodic features are used to combine with the text features for speech synthesis to obtain synthesized speech of the target text that has the predicted prosodic features.

[0007] In the above scheme, the feature extraction module is further configured to perform word segmentation on the target text to obtain multiple words included in the target text; for each word segment, obtain the phoneme information of the word segment, and encode the phoneme information to convert the phoneme information into phoneme features; combine the phoneme features of each word segment to obtain the text features of the target text.

[0008] In the above scheme, the sampling module is further configured to generate first random data conforming to the first data distribution type and construct the first target distribution based on the first random data before sampling initial prosodic features for prosodic prediction from the first target distribution and sampling noise for prosodic prediction from the second target distribution; generate second random data conforming to the second data distribution type and construct the second target distribution based on the second random data.

[0009] In the above scheme, the prosody prediction module is further configured to acquire prosody control information for prosody prediction before performing prosody prediction on the target text based on the text features, the initial prosody features, and the noise to obtain the predicted prosody features of the target text; the prosody prediction module is further configured to combine the text features and the prosody control information to obtain combined features; and perform prosody prediction on the target text based on the combined features, the initial prosody features, and the noise to obtain the predicted prosody features of the target text.

[0010] In the above scheme, the prosody prediction includes M rounds of prosody prediction; the sampling module is further configured to sample noise for each round of prosody prediction from the second target distribution for each round of prosody prediction in the M rounds of prosody prediction; the prosody prediction module is further configured to perform prosody prediction on the target text based on the text features, the initial prosody features, and the noise used for the first round of prosody prediction in the M rounds of prosody prediction, to obtain the intermediate predicted prosody features of the first round of prosody prediction; for the M rounds of prosody prediction... In the m-th round of prosodic prediction, based on the text features, the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction, and the noise used for the m-th round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the m-th round of prosodic prediction; m is traversed to obtain the intermediate predicted prosodic features of the M-th round of prosodic prediction, and the intermediate predicted prosodic features of the M-th round of prosodic prediction are used as the predicted prosodic features of the target text; wherein, M and m are integers greater than 0, and M is greater than or equal to m.

[0011] In the above scheme, the prosody prediction module is further configured to predict the noise to be removed from the initial prosody features based on the text features and the initial prosody features, to obtain the noise to be removed; subtract the initial prosody features from the noise to be removed, and determine the predicted prosody features of the target text based on the subtraction result and the noise.

[0012] In the above scheme, the first target distribution conforms to a first data distribution type, and the sampling module is further configured to randomly sample first sampled data conforming to the first data distribution type from the first target distribution, and use the first sampled data as the initial prosodic feature for prosodic prediction; the second target distribution conforms to a second data distribution type, and the sampling module is further configured to randomly sample second sampled data conforming to the second data distribution type from the second target distribution, and use the second sampled data as the noise for prosodic prediction.

[0013] In the above scheme, the prosody prediction module is further used to obtain a prosody prediction model for prosody prediction; based on the text features, the initial prosody features, and the noise, the prosody prediction model is invoked to perform prosody prediction on the target text to obtain the predicted prosody features of the target text.

[0014] In the above scheme, the prosody prediction module is further configured to acquire an initial prosody prediction model, and acquire sample text for training the initial prosody prediction model, as well as the target prosody features of the sample text; perform feature extraction on the sample text to obtain sample text features, and sample noise is sampled from a third target distribution; based on the sample text features, the target prosody features, and the sample noise, the initial prosody prediction model is used to predict the noise to be removed, and the predicted noise to be removed is used to determine the prosody prediction result of the initial prosody prediction model for the sample text; based on the predicted noise to be removed, the model parameters of the initial prosody prediction model are updated to obtain the prosody prediction model used for prosody prediction.

[0015] In the above scheme, the prosody prediction module is further configured to determine the gradient of the initial prosody prediction model based on the predicted noise to be removed; and update the model parameters of the initial prosody prediction model based on the gradient.

[0016] In the above scheme, the prosody prediction module is further used to perform speech synthesis on the target text based on the text features and the predicted prosody features to obtain the target synthesized speech of the target text.

[0017] This application also provides an electronic device, including: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the prosody prediction method provided in the embodiments of this application.

[0018] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs, which, when executed by a processor, implement the prosody prediction method provided in this application.

[0019] This application also provides a computer program product, including computer-executable instructions or a computer program, which, when executed by a processor, implements the prosody prediction method provided in this application.

[0020] The embodiments of this application have the following beneficial effects: By applying the above embodiments of this application, feature extraction is first performed on the target text to obtain the text features of the target text. Then, initial prosodic features for prosodic prediction are sampled from the first target distribution, and noise for prosodic prediction is sampled from the second target distribution. Based on the text features, the initial prosodic features, and the noise, prosodic prediction is performed on the target text to obtain the predicted prosodic features of the target text. In this way, the predicted prosodic features and the text features can be combined for speech synthesis to obtain the synthesized speech of the target text with the predicted prosodic features.

[0021] Here, because the initial prosodic features and noise sampled during prosodic prediction processing are random, the initial prosodic features and noise sampled are different between different rounds of prosodic prediction processing, resulting in different predicted prosodic features. Thus, the prosodic prediction method provided in this application can predict different predicted prosodic features for the same text, enriching the diversity of predicted prosodic features of the text, thereby enabling the synthesized speech obtained from speech synthesis based on predicted prosodic features to possess diversity. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of the prosody prediction system 100 provided in an embodiment of this application; Figure 2 This is a schematic diagram of the structure of the electronic device 500 that implements the prosody prediction method according to an embodiment of this application; Figure 3 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application; Figure 4 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application; Figure 5 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application; Figure 6 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application; Figure 7 This is a schematic diagram illustrating the application of the prosody prediction model provided in the embodiments of this application; Figure 8 This is a schematic diagram illustrating the training of the prosody prediction model provided in an embodiment of this application; Figure 9 This is a schematic diagram illustrating the application of the prosody prediction model provided in the embodiments of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0025] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0026] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0027] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0028] 1) Client: An application running in a terminal that provides various services, such as a client that supports prosody prediction.

[0029] 2) In response to, used to indicate the conditions or states on which the operation performed depends. When the conditions or states on which it depends are met, one or more operations performed may be performed in real time or with a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations are performed.

[0030] 3) Phoneme: The smallest unit of speech based on the natural properties of speech, which exists as the basic modeling unit in speech synthesis methods.

[0031] 4) Prosodic features, in connected speech, include not only the timbre characteristics of vowels and consonants arranged in chronological order, but also features such as pitch, intensity, duration, and their interrelationships. The prosodic features of speech are what is commonly referred to as intonation and rhythm. Specifically, prosodic features manifest as: tone at the syllable level, stress at the syllable combination level, length of sounds at the phoneme level, and intonation at the sentence level.

[0032] 5) Speech synthesis, also known as text-to-speech (TTS) technology, is a technology that generates artificial speech through mechanical and electronic methods. TTS technology converts text information generated by the computer itself or input from external sources into fluent, understandable spoken Chinese output.

[0033] 6) Encoding is the process of converting information from one form or format to another. Decoding is the reverse process of encoding.

[0034] This application provides a prosodic prediction method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can enrich the diversity of predicted prosodic features of text, enabling the synthesized speech obtained by speech synthesis based on predicted prosodic features to be diverse. These will be described in detail below.

[0035] It should be noted that when the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0036] The prosody prediction system provided in the embodiments of this application is described below. See also Figure 1 , Figure 1 This is a schematic diagram of the architecture of the prosody prediction system 100 provided in the embodiments of this application. In order to support an exemplary application, the terminal (terminal 400-1 is shown as an example) connects to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two, and data transmission is achieved using wireless or wired links.

[0037] The terminal (e.g., 400-1) is used to send a prosodic prediction request for the target text to the server 200 in response to a prosodic prediction instruction for the target text. Server 200 is configured to receive a prosodic prediction request for target text sent by a terminal; in response to the prosodic prediction request, acquire the target text and extract features from the target text to obtain text features of the target text; sample initial prosodic features for prosodic prediction from a first target distribution and sample noise for prosodic prediction from a second target distribution; and perform prosodic prediction on the target text based on the text features, initial prosodic features, and noise to obtain predicted prosodic features of the target text. In some embodiments, when speech synthesis of target text is required, the user can trigger a speech synthesis instruction for the target text on the terminal. At this time, the terminal (e.g., 400-1) is also used to send a speech synthesis request for the target text to the server 200; the server 200 is also used to receive the speech synthesis request for the target text; in response to the speech synthesis request, it performs speech synthesis based on the text features and predicted prosodic features of the target text to obtain the synthesized speech of the target text with predicted prosodic features; it returns the synthesized speech of the target text to the terminal; the terminal (e.g., 400-1) is also used to receive the synthesized speech of the target text returned by the server 200; and it plays the synthesized speech of the target text.

[0038] In some embodiments, the prosody prediction method provided in this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. The prosody prediction method provided in this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, assisted driving, games, audio and video, etc.

[0039] In some embodiments, the electronic device implementing the prosody prediction method provided in this application can be various types of terminals or servers. The server (e.g., server 200) can be a standalone physical server, a server cluster consisting of multiple physical servers, or a distributed system. The terminal (e.g., terminal 400-1) can be a smartphone, tablet computer, laptop computer, desktop computer, intelligent voice interaction device (e.g., smart speaker), smart home appliance (e.g., smart TV), smartwatch, in-vehicle terminal, wearable device, virtual reality (VR) device, etc., but is not limited thereto. The terminal and server can be directly or indirectly connected via wired or wireless communication, and this application does not impose any limitations on this.

[0040] In some embodiments, the prosody prediction method provided in this application can be implemented using cloud technology. Cloud technology refers to a hosting technology that unifies a series of resources such as hardware, software, and networks within a wide area network or local area network to realize data computation, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology applied based on the cloud computing business model. It can form a resource pool, be used on demand, and is flexible and convenient. Cloud computing technology will become an important support. The backend services of the technology network system require a large amount of computing and storage resources. As an example, the server (e.g., server 200) can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0041] In some embodiments, multiple servers can form a blockchain, with each server being a node on the blockchain. Information connections can exist between each node in the blockchain, allowing for information transmission between them. The data related to the prosody prediction method provided in this application (e.g., target text, predicted prosodic features of the target text, synthesized speech of the target text, etc.) can be stored on the blockchain.

[0042] In some embodiments, the terminal or server can implement the prosody prediction method provided in this application by running a computer program. For example, the computer program can be a native program or software module in an operating system; it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run; it can also be a mini-program, i.e., a program that only needs to be downloaded to a browser environment to run; or it can be a mini-program that can be embedded in any APP. In short, the above-mentioned computer program can be any form of application, module, or plugin.

[0043] The following describes an electronic device for implementing the prosody prediction method provided in an embodiment of this application. See also... Figure 2 , Figure 2This is a schematic diagram of the structure of an electronic device 500 implementing the prosody prediction method provided in this application embodiment. The electronic device 500 provided in this application embodiment can be a terminal or a server. The electronic device 500 provided in this application embodiment includes: at least one processor 510, a memory 550, at least one network interface 520, and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It is understood that the bus system 540 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 540 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 2 The general labeled all buses as Bus System 540.

[0044] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0045] User interface 530 includes one or more output devices 531 that enable the presentation of media content, including one or more speakers and / or one or more visual displays. User interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0046] Memory 550 may be removable, non-removable, or a combination thereof. Memory 550 may include one or more storage devices physically located away from processor 510. Memory 550 may include volatile memory or non-volatile memory, or both. Non-volatile memory may be read-only memory (ROM), and volatile memory may be random access memory (RAM). The memory 550 described in this application embodiment is intended to include any suitable type of memory.

[0047] In some embodiments, memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0048] Operating system 551 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 552 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. Presentation module 553 is used to enable the presentation of information (e.g., user interface for operating peripheral devices and displaying content and information) via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with user interface 530. The input processing module 554 is used to detect and translate one or more user inputs or interactions from one or more input devices 532.

[0049] In some embodiments, the prosody prediction device provided in this application can be implemented in software. Figure 2 A prosody prediction device 555 stored in memory 550 is shown. It can be software in the form of programs and plug-ins, including the following software modules: feature extraction module 5551, sampling module 5552 and prosody prediction module 5553. These modules are logical and can therefore be arbitrarily combined or further split according to the functions they implement. The functions of each module will be described below.

[0050] The prosodic prediction method provided in the embodiments of this application is described below. In some embodiments, the prosodic prediction method provided in the embodiments of this application can be implemented by various electronic devices. For example, it can be implemented by a terminal alone, by a server alone, or by a terminal and a server working together. Taking terminal implementation as an example, see [link to documentation]. Figure 3 , Figure 3 This is a flowchart illustrating the prosody prediction method provided in this application embodiment. The prosody prediction method provided in this application embodiment includes: Step 101: The terminal extracts features from the target text to obtain the text features of the target text.

[0051] In step 101, the terminal may be equipped with a client, such as a client that supports prosodic prediction. The terminal runs this client, and the user can trigger a prosodic prediction command for the target text through the client. In response to the prosodic prediction command, the terminal obtains the target text to be predicted. This target text can be set by the user as needed, generated based on the context (e.g., generated by a smart voice assistant based on the user's voice), or it can be existing text (e.g., audiobooks). For example, the target text could be: "Hello," or "It's nice to meet you." After obtaining the target text, the terminal first performs feature extraction on the target text to obtain its text features.

[0052] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application. Figure 3 Step 101 shown can be achieved through Figure 4 Steps 1011-1013 shown are implemented as follows: Step 1011, the target text is segmented to obtain multiple segments included in the target text; Step 1012, for each segment, the phoneme information of the segment is obtained and the phoneme information is encoded to convert the phoneme information into phoneme features; Step 1013, the phoneme features of each segment are combined to obtain the text features of the target text.

[0053] Here, when extracting features from the target text, phoneme-level features can be extracted. In practice, this can be achieved through a pre-built text encoder. This text encoder can be built based on at least one layer of Transformer network. The process of extracting features from the target text includes: In step 1011, the target text can be segmented to obtain multiple segments. Specifically, the target text can first be regularized to obtain standard text, and then segmented to obtain multiple segments. In step 1012, 1) for each segment, the phoneme information of the segment is obtained, specifically the phoneme information of each character in the segment. This phoneme information represents the pronunciation of the corresponding segment, such as the pinyin, phonetic symbols, or any other pronunciation symbols suitable for representing the pronunciation of the short phrase. 2) For each segment, the phoneme information of the segment is encoded to convert the phoneme information of the segment into phoneme features. In step 1013, the phoneme features of each word in the target text are combined to obtain the text features of the target text. Specifically, the phoneme features of each word can be concatenated to obtain concatenated phoneme features, which are then used as the text features of the target text. This concatenation process may include adding, multiplying, etc., the phoneme features of each word.

[0054] It should be noted that prosodic features are the prosodic characteristics of the phonemes in the target text, and each phoneme has corresponding prosodic features. These prosodic features can be represented by dimensions such as fundamental frequency, energy, and duration. In the embodiments of this application, text features can be understood as text representations, and prosodic features can be understood as prosodic representations.

[0055] Step 102: Sample initial prosodic features for prosodic prediction from the first target distribution, and sample noise for prosodic prediction from the second target distribution.

[0056] In step 102, the terminal can first acquire a first target distribution and a second target distribution. Then, random sampling is performed from the first target distribution to obtain initial prosodic features for prosodic prediction; and random sampling is performed from the second target distribution to obtain noise for prosodic prediction. In practical applications, the first target distribution can conform to a first data distribution type, meaning the first target distribution includes multiple random data points conforming to the first data distribution type, and the second target distribution can conform to a second data distribution type, meaning the second target distribution includes multiple random data points conforming to the second data distribution type. The first and second data distribution types can be the same or different. The data distribution type (first or second) can be a normal distribution, a standard normal distribution, etc.

[0057] In some embodiments, the terminal can construct a first target distribution by generating first random data conforming to a first data distribution type and constructing the first target distribution based on the first random data; correspondingly, the terminal can construct a second target distribution by generating second random data conforming to a second data distribution type and constructing the second target distribution based on the second random data. Here, the random data can be generated using a random data generation algorithm.

[0058] In some embodiments, when a target distribution (a first target distribution or a second target distribution) conforms to a certain target data distribution type (a first data distribution type or a second data distribution type), during random sampling, sampled data conforming to that target data distribution type can be sampled from the target distribution, and this sampled data can be used as the sampling result. That is, if the first target distribution conforms to the first data distribution type, the terminal can sample initial prosodic features from the first target distribution in the following manner: randomly sample first sampled data conforming to the first data distribution type from the first target distribution, and use the first sampled data as the initial prosodic features for prosodic prediction; correspondingly, if the second target distribution conforms to the second data distribution type, the terminal can sample noise from the second target distribution in the following manner: randomly sample second sampled data conforming to the second data distribution type from the second target distribution, and use the second sampled data as noise for prosodic prediction.

[0059] Here, because the initial prosodic features and noise sampled during prosodic prediction processing are random, the initial prosodic features and noise sampled are different between different rounds of prosodic prediction processing, resulting in different predicted prosodic features. Thus, by executing the prosodic prediction method provided in this application embodiment multiple times on the same text, different predicted prosodic features can be obtained for the same text, enriching the diversity of predicted prosodic features of the text, thereby enabling the synthesized speech obtained from speech synthesis based on predicted prosodic features to possess diversity.

[0060] Step 103: Based on text features, initial prosodic features, and noise, perform prosodic prediction on the target text to obtain the predicted prosodic features of the target text.

[0061] The predicted prosodic feature is used to combine with text features for speech synthesis, resulting in synthesized speech of the target text with the predicted prosodic feature.

[0062] In step 103, after obtaining the text features of the target text, the initial prosodic features for prosodic prediction, and the noise, prosodic prediction is performed on the target text based on the text features, the initial prosodic features, and the noise to obtain the predicted prosodic features of the target text.

[0063] In some embodiments, before performing prosodic prediction on the target text, the terminal can also obtain prosodic control information (or prosodic control conditions) for prosodic prediction. This prosodic control information can be pre-set or generated based on the context and content of the target text. The prosodic control information may include timbre, pitch, intensity, and emotion (e.g., happiness, sadness). Based on this, the terminal can perform prosodic prediction on the target text using text features, initial prosodic features, and noise in the following manner to obtain the predicted prosodic features of the target text: combining the text features and prosodic control information to obtain combined features; and performing prosodic prediction on the target text based on the combined features, initial prosodic features, and noise to obtain the predicted prosodic features of the target text. Here, when combining the text features and prosodic control information, the text features and prosodic control information can be concatenated, and the result of the concatenation process can be used as the combined feature. This concatenation process may include adding or multiplying the text features and prosodic control information.

[0064] In some embodiments, prosodic prediction is a process of denoising the initial prosodic features. Therefore, when performing prosodic prediction, what is actually predicted is the noise to be removed from the initial prosodic features. Based on this, see [link to relevant documentation]. Figure 5 , Figure 5 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application. Figure 3 Step 103 shown can also be achieved through Figure 5 Steps 1031-1032 shown are implemented as follows: Step 1031, based on text features and initial prosodic features, the noise to be removed is predicted on the initial prosodic features to obtain the noise to be removed; Step 1032, the initial prosodic features are subtracted from the noise to be removed, and based on the subtraction result and the noise, the predicted prosodic features of the target text are determined.

[0065] In some embodiments, prosodic prediction includes M-round prosodic prediction; correspondingly, see Figure 6 , Figure 6 This is a flowchart illustrating the prosody prediction method provided in the embodiments of this application. Figure 3 Step 102 shown can be achieved through Figure 6 Step 1021 shown implements the following: initial prosodic features for prosodic prediction are sampled from the first target distribution, and noise for each round of prosodic prediction in the M rounds is sampled from the second target distribution. Based on this, Figure 3 Step 103 shown can also be achieved through Figure 6 Steps 1033-1035 are implemented as follows: Step 1033, for the first round of prosodic prediction in M ​​rounds of prosodic prediction, based on text features, initial prosodic features, and noise used for the first round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the first round of prosodic prediction; Step 1034, for the m-th round of prosodic prediction in M ​​rounds of prosodic prediction, based on text features, intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction, and noise used for the m-th round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the m-th round of prosodic prediction; Step 1035, m is iterated to obtain the intermediate predicted prosodic features of the M-th round of prosodic prediction, and the intermediate predicted prosodic features of the M-th round of prosodic prediction are used as the predicted prosodic features of the target text; where M and m are integers greater than 0, and M is greater than or equal to m.

[0066] In practical applications, this prosody prediction includes M rounds of prosody prediction. In the embodiments of this application, for each round of prosody prediction in the M rounds, noise for that round of prosody prediction is sampled from the second target distribution; that is, each round of prosody prediction has its own noise for that round of prosody prediction. Based on this, the processing procedure for each round of prosody prediction in the M rounds of prosody prediction is described as follows: For the first round of prosodic prediction in M ​​rounds: based on text features, initial prosodic features, and noise used in the first round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the first round of prosodic prediction. Specifically, firstly, the text features, initial prosodic features, and noise used in the first round of prosodic prediction are concatenated to obtain concatenated features. Then, prosodic prediction is performed on the target text based on the concatenated features to obtain the intermediate predicted prosodic features of the first round of prosodic prediction.

[0067] For the m-th round of prosodic prediction in M-round prosodic prediction: Based on text features, intermediate predicted prosodic features from the (m-1)-th round of prosodic prediction, and noise used for the m-th round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the m-th round. Similarly, the text features, intermediate predicted prosodic features from the (m-1)-th round of prosodic prediction, and noise used for the m-th round of prosodic prediction can be concatenated to obtain concatenated features. Then, prosodic prediction is performed on the target text based on the concatenated features to obtain the intermediate predicted prosodic features of the m-th round of prosodic prediction. Thus, by traversing m, the intermediate predicted prosodic features of the M-th round of prosodic prediction are obtained, and the intermediate predicted prosodic features of the M-th round of prosodic prediction are used as the predicted prosodic features of the target text. Here, the intermediate predicted features of the next round can be calculated from the intermediate predicted prosodic features of the previous round. Therefore, by using the M-round cascaded prosodic prediction method, the accuracy of prosodic prediction is improved, and the expressiveness of the predicted prosodic features and the final synthesized speech is enhanced.

[0068] In practical applications, for the first round of prosodic prediction in M ​​rounds of prosodic prediction: based on text features and initial prosodic features, the noise to be removed from the initial prosodic features is predicted to obtain the first round of noise to be removed; the initial prosodic features are subtracted from the first round of noise to be removed, and based on the subtraction result and the noise, the intermediate predicted prosodic features of the first round of prosodic prediction are determined. For the m-th round of prosodic prediction in M ​​rounds of prosodic prediction: based on text features and the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction, the noise to be removed from the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction is predicted to obtain the (m-1)-th round of noise to be removed; the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction are subtracted from the (m-1)-th round of noise to be removed, and based on the subtraction result and the noise, the intermediate predicted prosodic features of the m-th round of prosodic prediction are determined. By iterating through m, the intermediate predicted prosodic features of the Mth round of prosodic prediction are obtained, and these intermediate predicted prosodic features are used as the predicted prosodic features of the target text. In this way, through M rounds of cascaded prosodic prediction, a gradual denoising process is achieved on the initial prosodic features, making the obtained predicted prosodic features more accurate, thereby improving the expressiveness of the predicted prosodic features and the final synthesized speech.

[0069] In some embodiments, the terminal can predict the prosody of the target text based on text features, initial prosodic features, and noise in the following way to obtain the predicted prosodic features of the target text: obtaining a prosodic prediction model for prosodic prediction; calling the prosodic prediction model to predict the prosody of the target text based on text features, initial prosodic features, and noise to obtain the predicted prosodic features of the target text. Here, the prosodic prediction model can be based on a neural network, such as a deep neural network or a convolutional neural network. In the embodiments of this application, the prosodic prediction model is built with a diffusion probability model as its core. In this case, the number of rounds M is equal to the total number of diffusion time steps T of the diffusion probability model (T is a positive integer). Each m value corresponds to a time step t, which belongs to the time step sequence {T, T-1, ..., 2, 1}. As m increases, the number of time steps t decreases until it reaches 1. Based on this, when performing each round of prosodic prediction, it is also necessary to combine the time step t for prosodic prediction. Specifically, the prosodic prediction model can predict the predicted prosodic representation through the following formula (I): ;(one) Where c is the input condition obtained by combining text representation with prosodic control conditions (i.e., the prosodic control information mentioned above); The predicted prosodic representation obtained from the previous time step (i.e., the intermediate predicted prosodic features of the (m-1)th round of prosodic prediction); z is the noise sampled from the second target distribution (e.g., the standard normal distribution) at each time step t; fixed parameters , The noise to be removed is calculated. After repeating the calculation T times according to the above formula (I), the final predicted prosodic representation is obtained.

[0070] In addition, since the prosody prediction process of the prosody prediction model includes M rounds of prosody prediction, it is possible to sample N rounds of prosody prediction (N is a positive integer less than M) from the M rounds of prosody prediction for training. This improves the training efficiency of the model and reduces the waste of computing resources.

[0071] In practice, the prosody prediction model is pre-trained. The training process of the prosody prediction model will be explained below.

[0072] In some embodiments, the terminal may train a prosody prediction model as follows: obtain an initial prosody prediction model, and obtain sample text for training the initial prosody prediction model, as well as the target prosody features of the sample text; extract features from the sample text to obtain sample text features; based on the sample text features and the target prosody features, predict the noise to be removed using the initial prosody prediction model to obtain the predicted noise to be removed, which is used to determine the prosody prediction result of the initial prosody prediction model for the sample text; based on the predicted noise to be removed, update the model parameters of the initial prosody prediction model to obtain a prosody prediction model for prosody prediction.

[0073] In some embodiments, the terminal may update the model parameters of the initial prosodic prediction model based on the predicted noise to be removed by: determining the gradient of the initial prosodic prediction model based on the predicted noise to be removed; and updating the model parameters of the initial prosodic prediction model based on the gradient.

[0074] Here, during the model training phase, the training objective of the prosody prediction model is to maximize the variational lower bound of the log-likelihood of the prosody representation distribution. To achieve the effect of training the prosody prediction model with this training objective, in the actual training process, the prosody prediction model processes the training data (including sample text features and target prosody features). Specifically, the prosody prediction model predicts the noise to be removed based on the training data, obtains the predicted noise to be removed, and then calculates the gradient of the initial prosody prediction model based on the predicted noise to be removed. The model parameters of the initial prosody prediction model are then updated based on the gradient until convergence, thus completing the training.

[0075] In practical applications, when the initial prosodic prediction model is constructed based on the diffusion probability model, the gradient of the initial prosodic prediction model can be determined by using the following formula (ii) based on the predicted noise to be removed: ;(two) Where c is the input condition obtained by combining the sample text representation with the prosodic control condition (i.e., the prosodic control information mentioned above); t is the current time step obtained by uniformly sampling on the set of positive integers between 1 and T. This is noise sampled from a standard normal distribution; The noise to be removed is for the prediction at the current time step. For the target prosodic representation; while fixed parameters , , This represents the diffusion noise to be added at the current time step t.

[0076] In practical applications, when speech synthesis is required, the target text can be synthesized based on text features and predicted prosodic features to obtain the target synthesized speech. This speech synthesis process can be implemented using a pre-trained speech synthesis model. Specifically, the acoustic decoder of the speech synthesis model can decode the text features and predicted prosodic features to obtain the target synthesized speech.

[0077] By applying the above embodiments of this application, feature extraction is first performed on the target text to obtain the text features of the target text. Then, initial prosodic features for prosodic prediction are sampled from the first target distribution, and noise for prosodic prediction is sampled from the second target distribution. Based on the text features, the initial prosodic features, and the noise, prosodic prediction is performed on the target text to obtain the predicted prosodic features of the target text. In this way, the predicted prosodic features and the text features can be combined for speech synthesis to obtain the synthesized speech of the target text with the predicted prosodic features.

[0078] Here, because the initial prosodic features and noise sampled during prosodic prediction processing are random, the initial prosodic features and noise sampled are different between different rounds of prosodic prediction processing, resulting in different predicted prosodic features. Thus, the prosodic prediction method provided in this application can predict different predicted prosodic features for the same text, enriching the diversity of predicted prosodic features of the text, thereby enabling the synthesized speech obtained from speech synthesis based on predicted prosodic features to possess diversity.

[0079] The following example, using a prosody prediction model constructed based on a diffusion probability model, illustrates an exemplary application of this application in a real-world scenario.

[0080] First, let's explain the prosodic prediction methods provided in related technologies. These technologies presuppose that the prosodic representation of real human speech follows a simple unimodal Laplace or Gaussian distribution, and then train the prosodic prediction model by optimizing the prediction error. The prosodic prediction model employs a traditional deterministic prediction model, meaning that under the same prosodic control conditions (i.e., the aforementioned prosodic control information), the output predicted prosodic representation remains unchanged for a single input text. Therefore, these technologies have the following drawbacks: 1) The presupposition of the prosodic representation distribution of human speech is overly simplified, deviating from reality, leading to overly smooth predicted prosodic representations and ultimately reducing the expressiveness of the synthesized speech; 2) The training method of directly optimizing the prediction error causes the prosodic prediction model to overfit to the sample texts in the training data, resulting in poor generalization performance; 3) The deterministic model prediction results in a lack of variation in the predicted prosodic representation of a single input text, leading to overly simplistic synthesized speech based on this predicted prosodic representation for that input text.

[0081] Based on this, embodiments of this application provide a prosodic prediction method to at least solve the aforementioned problems. Specifically, embodiments of this application provide a prosodic prediction method based on a diffusion probability model, which is geared towards speech synthesis tasks. Specifically, the target text and prosodic control conditions are used as inputs to the prosodic prediction model, which predicts and outputs the corresponding predicted prosodic representation. This predicted prosodic representation can be used to control the prosodic variations of synthesized speech generated by the speech synthesis task, thereby improving the expressiveness and diversity of the synthesized speech. Based on this, the prosody prediction method provided in this application addresses the following issues: 1) The diffusion probability model does not pre-define the original distribution of prosody representations. Instead, it trains the model by maximizing the variational lower bound of the log-likelihood of the prosody representation distribution and performs prosody prediction in a progressively denoised manner. This allows for the modeling of more complex prosody representation distributions, avoids oversmoothing, and improves the expressiveness of predicted prosody representations and the final synthesized speech. 2) As a generative model, the diffusion probability model randomly samples initial prosody features and noise from a normal distribution when predicting prosody representations. This makes the initial prosody features and noise sampled in each round random, resulting in different predicted prosody representations obtained in different rounds. That is, for the same text, different predicted prosody representations can be obtained by executing the prosody prediction method provided in this application multiple times, enriching the diversity of predicted prosody representations of the text and thus improving the diversity of synthesized speech.

[0082] In practical applications, the prosody prediction method provided in this application can be applied to content generation scenarios that rely on speech synthesis, such as intelligent voice assistants and automatic generation of electronic audiobooks. Specifically, see... Figure 7 , Figure 7This is a schematic diagram illustrating the application of the prosodic prediction model provided in this application embodiment. Here, the prosodic prediction model provided in this application embodiment can be combined with a speech synthesis framework including a text encoder and an acoustic decoder to form a structure such as... Figure 7 The speech synthesis service process shown includes: (1) The service caller provides the input text and prosodic control conditions (such as timbre, emotion, etc., which are covered by the training data); (2) The text encoder obtains the text representation based on the input text; (3) The prosodic prediction model performs prosodic prediction based on the text representation and prosodic control conditions to obtain the predicted prosodic representation; (4) The acoustic decoder decodes the synthesized speech of the input text based on the text representation and the predicted prosodic representation. The content of the synthesized speech is the input text, and the prosodic of the synthesized speech is regulated by the predicted prosodic representation.

[0083] The prosodic prediction method provided in this application mainly involves a text encoder and a prosodic prediction model. Specifically, 1) the text encoder can employ a multi-layer feedforward Transformer network pre-trained with the speech synthesis framework, responsible for encoding the input text into phoneme-level representations (i.e., text representations). During model training, the parameters of the text encoder can be kept frozen. 2) The prosodic prediction model employs a diffusion probability model using a non-causal WaveNet as a denoiser. Given text representations and prosodic control conditions, it is responsible for modeling the prosodic representation distribution and providing the function of prediction based on the modeled prosodic representation distribution, thereby obtaining the predicted prosodic representation. Specifically, the main parameters of the diffusion probability model include: a) preset fixed parameters: a1) the total number of time steps T in the diffusion process (i.e., the number of rounds M in the above prosodic prediction, which is a positive integer); a2) a diffusion noise sequence {β1, ..., β2} of length T to be added during the diffusion process. T b) Trainable parameters: Neural network parameters θ in the non-causal WaveNet-based denoiser.

[0084] The training process of the prosody prediction model provided in the embodiments of this application will be described next. See [link to documentation]. Figure 8 , Figure 8 This is a schematic diagram illustrating the training of the prosodic prediction model provided in this application embodiment. Here, during the model training phase, the training objective of the prosodic prediction model is to maximize the variational lower bound of the log-likelihood of the prosodic representation distribution. To achieve the effect of training the prosodic prediction model with this training objective, in the actual training process, the prosodic prediction model is used to train the training data (including sample text, prosodic control conditions, and the target prosodic representation of the sample text). The processing involves using the denoiser in the diffusion probability model to predict the noise to be removed at the current time step t based on the training data, outputting the predicted noise to be removed, and then calculating the gradient based on the following gradient calculation formula (II), thereby updating the trainable parameters θ of the prosody prediction model through the gradient until convergence, which completes the training.

[0085] ;(two) Where c is the input condition obtained by combining the sample text in the training dataset, which is encoded into a sample text representation by the text encoder, with the prosodic control condition; t is the current time step obtained by uniform sampling on the set of positive integers between 1 and T. Noise sampled from a third target distribution (such as a standard normal distribution); Here, the noise to be removed is predicted at the current time step by the denoiser in the diffusion probability model. To train the target prosodic representation of each phoneme in the pre-processed dataset, the prosodic representation consists of three dimensions: fundamental frequency, energy, and duration; while fixed parameters... , .

[0086] The application process of the prosody prediction model provided in the embodiments of this application will be described next. See [link to relevant documentation]. Figure 9 , Figure 9 This is a schematic diagram illustrating the application of the prosody prediction model provided in this application embodiment. Here, in the model inference application stage, the prosody prediction model starts from the initial prosody representation sampled from the standard normal distribution. Starting from this point, for each time step t in the diffusion time step sequence {T, T-1, ..., 1}, the prosodic representation is updated sequentially according to the following formula (I): ;(one) Where c is the input condition obtained by combining the text representation of the input text (i.e. the target text mentioned above) with the prosodic control condition; is the predicted prosodic representation obtained from the previous time step; z is the noise sampled from the second target distribution (such as the standard normal distribution) at each time step t; For noise to be removed; fixed parameters After repeating the calculation T times according to the above formula (I), the final predicted prosodic representation is obtained.

[0087] In practical applications, 1) the multi-layer feedforward Transformer model in the text encoder can be replaced by other neural network structures; 2) the non-causal WaveNet network of the diffusion probability model in the prosody prediction model can also be replaced by other neural network structures; 3) the sampling method of the current time step t during model training can be replaced by weighted sampling, where the weights can be dynamically updated; 4) the representation method of the current time step t during model training can also be replaced by upsampling the set of positive integers between 1 and T and dividing by T to obtain its decimal representation; 5) the components of the prosody representation can also be replaced by a combination of other types of acoustic features or implicit representations extracted by neural networks.

[0088] Applying the above embodiments of this application, 1) compared with related technologies, the embodiments of this application generally improve the fitting effect of the predicted prosodic representation distribution to the real distribution. The Jensen-Shannon Divergence distributions (within the interval (0,1), where a smaller value indicates a better fitting effect) on the three dimensions of prosodic prediction representation (fundamental frequency, energy, and duration) decrease from 0.199, 0.056, and 0.119 to 0.085, 0.055, and 0.056, respectively. 2) compared with related technologies, the embodiments of this application improve the expressiveness of the predicted prosodic representation and the synthesized speech controlled by it. 3) compared with related technologies, the embodiments of this application enrich the diversity of the predicted prosodic representation of a single text, enabling different predicted prosodic representations to be obtained through different rounds of prosodic prediction processing. This improves the problem that related technologies can only output the same predicted prosodic representation for a single text under the same conditions, thereby enhancing the diversity of synthesized speech.

[0089] The following description continues to illustrate the exemplary structure of the prosody prediction device 555 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the prosody prediction device 555 in the memory 550 may include: a feature extraction module 5551, used to extract features from the target text to obtain text features of the target text; a sampling module 5552, used to sample initial prosodic features for prosodic prediction from a first target distribution and sample noise for prosodic prediction from a second target distribution; and a prosodic prediction module 5553, used to perform prosodic prediction on the target text based on the text features, the initial prosodic features, and the noise to obtain predicted prosodic features of the target text; wherein, the predicted prosodic features are used to combine with the text features for speech synthesis to obtain synthesized speech of the target text having the predicted prosodic features.

[0090] In some embodiments, the feature extraction module 5551 is further configured to perform word segmentation on the target text to obtain multiple words included in the target text; for each word segment, obtain the phoneme information of the word segment, and encode the phoneme information to convert the phoneme information into phoneme features; combine the phoneme features of each word segment to obtain the text features of the target text.

[0091] In some embodiments, the sampling module 5552 is further configured to generate first random data conforming to a first data distribution type and construct the first target distribution based on the first random data before sampling initial prosodic features for prosodic prediction from the first target distribution and sampling noise for prosodic prediction from the second target distribution; generate second random data conforming to a second data distribution type and construct the second target distribution based on the second random data.

[0092] In some embodiments, the prosody prediction module 5553 is further configured to acquire prosody control information for prosody prediction before performing prosody prediction on the target text based on the text features, the initial prosody features, and the noise to obtain the predicted prosody features of the target text; the prosody prediction module 5553 is further configured to combine the text features and the prosody control information to obtain combined features; and perform prosody prediction on the target text based on the combined features, the initial prosody features, and the noise to obtain the predicted prosody features of the target text.

[0093] In some embodiments, the prosody prediction includes M rounds of prosody prediction; the sampling module 5552 is further configured to sample noise for each round of prosody prediction from the second target distribution for each round of prosody prediction in the M rounds of prosody prediction; the prosody prediction module 5553 is further configured to perform prosody prediction on the target text based on the text features, the initial prosody features, and the noise for the first round of prosody prediction in the M rounds of prosody prediction, to obtain the intermediate predicted prosody features of the first round of prosody prediction; for the... In the m-th round of prosodic prediction, based on the text features, the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction, and the noise used for the m-th round of prosodic prediction, prosodic prediction is performed on the target text to obtain the intermediate predicted prosodic features of the m-th round of prosodic prediction; m is traversed to obtain the intermediate predicted prosodic features of the M-th round of prosodic prediction, and the intermediate predicted prosodic features of the M-th round of prosodic prediction are used as the predicted prosodic features of the target text; wherein, M and m are integers greater than 0, and M is greater than or equal to m.

[0094] In some embodiments, the prosody prediction module 5553 is further configured to predict noise to be removed from the initial prosody features based on the text features and the initial prosody features, to obtain noise to be removed; subtract the initial prosody features from the noise to be removed, and determine the predicted prosody features of the target text based on the subtraction result and the noise.

[0095] In some embodiments, the first target distribution conforms to a first data distribution type, and the sampling module 5552 is further configured to randomly sample first sampled data conforming to the first data distribution type from the first target distribution, and use the first sampled data as the initial prosodic feature for prosodic prediction; the second target distribution conforms to a second data distribution type, and the sampling module 5552 is further configured to randomly sample second sampled data conforming to the second data distribution type from the second target distribution, and use the second sampled data as the noise for prosodic prediction.

[0096] In some embodiments, the prosody prediction module 5553 is further configured to obtain a prosody prediction model for prosody prediction; based on the text features, the initial prosody features, and the noise, call the prosody prediction model to perform prosody prediction on the target text to obtain the predicted prosody features of the target text.

[0097] In some embodiments, the prosody prediction module 5553 is further configured to: acquire an initial prosody prediction model; acquire sample text for training the initial prosody prediction model and target prosody features of the sample text; extract features from the sample text to obtain sample text features, and sample noise from a third target distribution; based on the sample text features, the target prosody features, and the sample noise, predict the noise to be removed using the initial prosody prediction model to obtain predicted noise to be removed, wherein the predicted noise to be removed is used to determine the prosody prediction result of the initial prosody prediction model for the sample text; and update the model parameters of the initial prosody prediction model based on the predicted noise to be removed to obtain the prosody prediction model used for prosody prediction.

[0098] In some embodiments, the prosody prediction module 5553 is further configured to determine the gradient of the initial prosody prediction model based on the predicted noise to be removed; and update the model parameters of the initial prosody prediction model based on the gradient.

[0099] In some embodiments, the prosody prediction module 5553 is further configured to perform speech synthesis on the target text based on the text features and the predicted prosody features to obtain the target synthesized speech of the target text.

[0100] By applying the above embodiments of this application, feature extraction is first performed on the target text to obtain the text features of the target text. Then, initial prosodic features for prosodic prediction are sampled from the first target distribution, and noise for prosodic prediction is sampled from the second target distribution. Based on the text features, the initial prosodic features, and the noise, prosodic prediction is performed on the target text to obtain the predicted prosodic features of the target text. In this way, the predicted prosodic features and the text features can be combined for speech synthesis to obtain the synthesized speech of the target text with the predicted prosodic features.

[0101] Here, because the initial prosodic features and noise sampled during prosodic prediction processing are random, the initial prosodic features and noise sampled are different between different rounds of prosodic prediction processing, resulting in different predicted prosodic features. Thus, the prosodic prediction method provided in this application can predict different predicted prosodic features for the same text, enriching the diversity of predicted prosodic features of the text, thereby enabling the synthesized speech obtained from speech synthesis based on predicted prosodic features to possess diversity.

[0102] This application also provides a computer program product, which includes computer-executable instructions or a computer program stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions or computer program from the computer-readable storage medium and executes the computer-executable instructions or computer program, causing the electronic device to perform the prosody prediction method provided in this application.

[0103] This application also provides a computer-readable storage medium storing computer-executable instructions or computer programs. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the prosody prediction method provided in this application.

[0104] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0105] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0106] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0107] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located in one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0108] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A prosody prediction method, characterized in that, The method includes: Feature extraction is performed on the target text to obtain the text features of the target text; Generate first random data that conforms to a first data distribution type, and construct a first target distribution based on the first random data; Generate second random data that conforms to the second data distribution type, and construct a second target distribution based on the second random data; From the first target distribution, first sampled data that conforms to the first data distribution type is randomly sampled, and the first sampled data is used as the initial prosodic features for prosodic prediction; From the second target distribution, second sampled data that conforms to the second data distribution type is randomly sampled, and the second sampled data is used as noise for prosody prediction; Based on the text features, the initial prosodic features, and the noise, the prosodic features of the target text are predicted to be obtained. The predicted prosodic features are used to combine with the text features for speech synthesis to obtain synthesized speech of the target text that has the predicted prosodic features.

2. The method as described in claim 1, characterized in that, The step of extracting features from the target text to obtain the text features of the target text includes: The target text is segmented to obtain multiple segments comprising the target text; For each word segment, the phoneme information of the word segment is obtained, and the phoneme information is encoded to convert the phoneme information into phoneme features; The phoneme features of each segmented word are combined to obtain the text features of the target text.

3. The method as described in claim 1, characterized in that, Before performing prosodic prediction on the target text based on the text features, the initial prosodic features, and the noise to obtain the predicted prosodic features of the target text, the method further includes: Obtain prosodic control information for prosodic prediction; The step of predicting the prosodic features of the target text based on the text features, the initial prosodic features, and the noise, to obtain the predicted prosodic features of the target text, includes: The text features and the prosody control information are combined to obtain combined features; Based on the combined features, the initial prosodic features, and the noise, the prosodic prediction of the target text is performed to obtain the predicted prosodic features of the target text.

4. The method as described in claim 1, characterized in that, The prosodic prediction includes M-round prosodic prediction; the method further includes: For each round of prosody prediction in the M-round prosody prediction, noise for the round prosody prediction is sampled from the second target distribution; The step of predicting the prosodic features of the target text based on the text features, the initial prosodic features, and the noise, to obtain the predicted prosodic features of the target text, includes: For the first round of prosodic prediction in the M rounds of prosodic prediction, based on the text features, the initial prosodic features, and the noise used for the first round of prosodic prediction, the target text is prosodic predicted to obtain the intermediate predicted prosodic features of the first round of prosodic prediction. For the m-th round of prosodic prediction in the M-th round of prosodic prediction, based on the text features, the intermediate predicted prosodic features of the (m-1)-th round of prosodic prediction, and the noise used for the m-th round of prosodic prediction, the target text is prosodic predicted to obtain the intermediate predicted prosodic features of the m-th round of prosodic prediction. Traverse the m to obtain the intermediate predicted prosodic features of the Mth round of prosodic prediction, and use the intermediate predicted prosodic features of the Mth round of prosodic prediction as the predicted prosodic features of the target text. Wherein, M and m are integers greater than 0, and M is greater than or equal to m.

5. The method as described in claim 1, characterized in that, The step of predicting the prosodic features of the target text based on the text features, the initial prosodic features, and the noise, to obtain the predicted prosodic features of the target text, includes: Based on the text features and the initial prosodic features, the noise to be removed is predicted on the initial prosodic features to obtain the noise to be removed; The initial prosodic features are subtracted from the noise to be removed, and the predicted prosodic features of the target text are determined based on the subtraction result and the noise.

6. The method as described in claim 1, characterized in that, The step of predicting the prosodic features of the target text based on the text features, the initial prosodic features, and the noise, to obtain the predicted prosodic features of the target text, includes: Obtain a prosodic prediction model for prosodic prediction; Based on the text features, the initial prosodic features, and the noise, the prosodic prediction model is invoked to predict the prosodic features of the target text, thereby obtaining the predicted prosodic features of the target text.

7. The method as described in claim 6, characterized in that, The process of obtaining the prosodic prediction model for prosodic prediction includes: Obtain an initial prosodic prediction model, and obtain sample texts for training the initial prosodic prediction model, as well as the target prosodic features of the sample texts; Feature extraction is performed on the sample text to obtain the sample text features, and sample noise is sampled from the third target distribution; Based on the sample text features, the target prosodic features, and the sample noise, the noise to be removed is predicted using the initial prosodic prediction model to obtain the predicted noise to be removed. The predicted noise to be removed is used to determine the prosodic prediction result of the initial prosodic prediction model for the sample text. Based on the predicted noise to be removed, the model parameters of the initial prosodic prediction model are updated to obtain the prosodic prediction model used for prosodic prediction.

8. The method as described in claim 7, characterized in that, The step of updating the model parameters of the initial prosodic prediction model based on the predicted noise to be removed includes: Based on the predicted noise to be removed, the gradient of the initial prosodic prediction model is determined; Based on the gradient, update the model parameters of the initial prosody prediction model.

9. The method as described in claim 1, characterized in that, The method further includes: Based on the text features and the predicted prosodic features, speech synthesis is performed on the target text to obtain the target synthesized speech of the target text.

10. A prosody prediction device, characterized in that, The device includes: The feature extraction module is used to extract features from the target text to obtain the text features of the target text; A sampling module is used to generate first random data that conforms to a first data distribution type, and construct a first target distribution based on the first random data; generate second random data that conforms to a second data distribution type, and construct a second target distribution based on the second random data; The sampling module is further configured to randomly sample first sampled data conforming to the first data distribution type from the first target distribution, and use the first sampled data as initial prosodic features for prosodic prediction; and randomly sample second sampled data conforming to the second data distribution type from the second target distribution, and use the second sampled data as noise for prosodic prediction. The prosody prediction module is used to predict the prosody of the target text based on the text features, the initial prosody features, and the noise, so as to obtain the predicted prosody features of the target text. The predicted prosodic features are used to combine with the text features for speech synthesis to obtain synthesized speech of the target text that has the predicted prosodic features.

11. The apparatus according to claim 10, characterized in that, The feature extraction module is further configured to perform word segmentation on the target text to obtain multiple words included in the target text; for each word segment, obtain the phoneme information of the word segment, and encode the phoneme information to convert the phoneme information into phoneme features; combine the phoneme features of each word segment to obtain the text features of the target text.

12. The apparatus according to claim 10, characterized in that, The prosody prediction module is further configured to obtain prosody control information for prosody prediction before performing prosody prediction on the target text based on the text features, the initial prosody features, and the noise to obtain the predicted prosody features of the target text. The prosody prediction module is further configured to combine the text features and the prosody control information to obtain combined features; and based on the combined features, the initial prosody features, and the noise, to perform prosody prediction on the target text to obtain the predicted prosody features of the target text.

13. The apparatus according to claim 10, characterized in that, The prosody prediction includes M rounds of prosody prediction; The sampling module is further configured to sample noise from the second target distribution for each round of prosody prediction in the M-round prosody prediction; The prosody prediction module is further configured to perform prosody prediction on the target text based on the text features, the initial prosody features, and the noise used for the first round of prosody prediction in the M rounds of prosody prediction, to obtain the intermediate predicted prosody features of the first round of prosody prediction. The prosody prediction module is further configured to perform prosody prediction on the target text for the m-th round of prosody prediction in the M-th round of prosody prediction, based on the text features, the intermediate predicted prosody features of the (m-1)-th round of prosody prediction, and the noise used for the m-th round of prosody prediction, to obtain the intermediate predicted prosody features of the m-th round of prosody prediction. The prosody prediction module is also used to traverse the m to obtain the intermediate predicted prosody features of the Mth round of prosody prediction, and use the intermediate predicted prosody features of the Mth round of prosody prediction as the predicted prosody features of the target text. Wherein, M and m are integers greater than 0, and M is greater than or equal to m.

14. An electronic device, characterized in that, The electronic device includes: Memory is used to store executable instructions for a computer; A processor, when executing computer-executable instructions stored in the memory, implements the prosody prediction method according to any one of claims 1 to 9.

15. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the prosody prediction method according to any one of claims 1 to 9 is implemented.

16. A computer program product comprising computer-executable instructions or a computer program, characterized in that, When the computer-executable instructions or computer program are executed by a processor, the prosody prediction method according to any one of claims 1 to 9 is implemented.

Images