A method and apparatus for training a speech unit sequence generation model

Abstract

This specification provides a training method and apparatus for a speech unit sequence generation model. The scheme may include: obtaining a sample speech unit sequence corresponding to a sample text; performing a masking operation on the sample speech unit sequence to obtain a first speech unit sequence; performing a length change operation on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation; performing a masking operation on the functional speech units in the second speech unit sequence to obtain a third speech unit sequence; and training a speech unit sequence generation model to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain a trained speech unit sequence generation model.

Description

[0001] This application claims priority to Chinese patent application No. 202511705986.7, filed on November 19, 2025, entitled “A Training Method and Apparatus for a Speech Unit Sequence Generation Model”. Technical Field

[0002] This application relates to the field of data processing technology, and in particular to a training method and apparatus for a speech unit sequence generation model. Background Technology

[0003] With the rapid development of artificial intelligence technology, text-to-speech (TTS) technology, which converts written text into audible speech, has become one of the most important branches of artificial intelligence. This technology can efficiently parse input text information and generate speech output, enriching the ways information is transmitted and enhancing the convenience for users to obtain information.

[0004] In practical applications, the quality of speech generation can significantly impact user experience. Therefore, improving the quality of speech generation is a pressing technical issue that needs to be addressed. Summary of the Invention

[0005] In view of the above, one or more embodiments of this specification provide a training method for a speech unit sequence generation model, a speech unit sequence generation method, a speech synthesis method, a training device for a speech unit sequence generation model, a speech unit sequence generation device, a speech synthesis device, a computing device, a computer-readable storage medium, and a computer program product to improve the quality of speech generation.

[0006] According to a first aspect of one or more embodiments of this specification, a method for training a speech unit sequence generation model is provided, comprising: Obtain the sample speech unit sequence corresponding to the sample text; The sample speech unit sequence is masked to obtain the first speech unit sequence; A length change operation is performed on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation. The functional speech units in the second speech unit sequence are masked to obtain the third speech unit sequence; Based on the sample text, the sample speech unit sequence, and the third speech unit sequence, the speech unit sequence generation model to be trained is trained to obtain the trained speech unit sequence generation model.

[0007] According to a second aspect of one or more embodiments of this specification, a method for generating a speech unit sequence is provided, comprising: Get the text to be processed; The text to be processed is input into the trained speech unit sequence generation model so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence; the first probability distribution for each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; The trained speech unit sequence generation model identifies functional speech units at each mask position in the preset speech unit sequence based on the first probability distribution. The functional speech unit is subjected to a length change inverse process to obtain the first processed speech unit sequence; Based on the first probability distribution, determine the speech information corresponding to the mask position that meets the preset filtering conditions in the first processed speech unit sequence, and obtain the second processed speech unit sequence. The mask positions in the second processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

[0008] According to a third aspect of one or more embodiments of this specification, a speech synthesis method is provided, comprising: Get the text to be processed; The text to be processed is input into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the method described in any one of claims 1-8; The predicted speech unit sequence is decoded to obtain the audio file of the text to be processed.

[0009] According to a fourth aspect of one or more embodiments of this specification, a training apparatus for a speech unit sequence generation model is provided, comprising: The first acquisition module is used to acquire the sample speech unit sequence corresponding to the sample text; The first operation module is used to perform a masking operation on the sample speech unit sequence to obtain the first speech unit sequence. The second operation module is used to perform a length change operation on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation. The third operation module is used to perform a masking operation on the functional speech units in the second speech unit sequence to obtain the third speech unit sequence. The training module is used to train the speech unit sequence generation model to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence, so as to obtain the trained speech unit sequence generation model.

[0010] According to a fifth aspect of one or more embodiments of this specification, a speech unit sequence generation apparatus is provided, comprising: The second acquisition module is used to acquire the text to be processed. The first input module is used to input the text to be processed into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution of each mask position in the preset speech unit sequence; the first probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; The recognition module is used to identify the functional speech units in each mask position of the preset speech unit sequence according to the first probability distribution by the trained speech unit sequence generation model. The inverse processing module is used to perform length-change inverse processing on the functional speech units to obtain a first processed speech unit sequence; The voice information determination module is used to determine the voice information corresponding to the mask position that meets the preset filtering conditions in the first processed voice unit sequence according to the first probability distribution, so as to obtain the second processed voice unit sequence. The prediction module is used to predict the mask positions in the second processed speech unit sequence to obtain the predicted speech unit sequence.

[0011] According to a sixth aspect of one or more embodiments of this specification, a speech synthesis apparatus is provided, comprising: The third acquisition module is used to acquire the text to be processed; The second input module is used to input the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the training method of the speech unit sequence generation model described in any embodiment of this specification; The decoding module is used to decode the predicted speech unit sequence to obtain the audio file of the text to be processed.

[0012] According to a seventh aspect of one or more embodiments of this specification, a computing device is provided, including a memory, a processor, and computer instructions stored in the memory and executable on the processor, wherein the processor executes the computer instructions to implement the steps of the method described above.

[0013] According to an eighth aspect of one or more embodiments of this specification, a computer-readable storage medium is provided that stores computer instructions which, when executed by a processor, implement the steps of the method described above.

[0014] According to a ninth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0015] One embodiment of this specification can achieve at least the following beneficial effects: This embodiment can obtain a sample speech unit sequence corresponding to sample text; perform a masking operation on the sample speech unit sequence to obtain a first speech unit sequence; perform a length modification operation on the first speech unit sequence to obtain a second speech unit sequence; perform a masking operation on the functional speech units in the second speech unit sequence to obtain a third speech unit sequence. Thus, a speech unit sequence generation model to be trained can be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence. During training, the model can predict the functional speech units in the third speech unit sequence, process the functional speech units to adjust the length of the third speech unit sequence, and then predict the speech units represented by the masked positions in the third speech unit sequence. This enables the trained speech unit sequence generation model to generate speech unit sequences of a length matching the text to be processed, improving the realism and naturalness of the generated speech unit sequences, thereby improving the quality of speech generation. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments or prior art of this specification, the drawings used in the description of the embodiments or prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of a training method for a speech unit sequence generation model provided in one embodiment of this specification. Figure 2 This is a flowchart illustrating a training method for a speech unit sequence generation model provided in one embodiment of this specification; Figure 3 This is a schematic diagram of a scenario for obtaining a third speech unit sequence provided in one embodiment of this specification; Figure 4 This is a schematic diagram of a scenario for training a speech unit sequence generation model according to an embodiment of this specification; Figure 5This is a flowchart of a speech unit sequence generation method provided in one embodiment of this specification; Figure 6 This is a flowchart of a speech synthesis method provided in one embodiment of this specification; Figure 7 This is a schematic diagram of the structure of a training device for a speech unit sequence generation model provided in one embodiment of this specification; Figure 8 This is a schematic diagram of the structure of a speech unit sequence generation device provided in one embodiment of this specification; Figure 9 This is a schematic diagram of the structure of a speech synthesis device provided in one embodiment of this specification; Figure 10 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0018] To enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this specification, and not all embodiments. Based on the embodiments in this specification, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of this specification.

[0019] This specification uses specific terms to describe embodiments thereof. Terms such as "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic associated with at least one embodiment of this specification. Therefore, it should be emphasized and noted that references to "an embodiment," "one embodiment," or "an alternative embodiment" in different locations throughout this specification do not necessarily refer to the same embodiment. Furthermore, those skilled in the art can combine and integrate the different embodiments or examples described herein, as well as the features of those different embodiments or examples, without contradiction.

[0020] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “an,” “an,” “the,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification includes any or all possible combinations of one or more associated listed items.

[0021] The terms “comprising,” “including,” or any other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitation, the presence of additional identical or equivalent elements in the process, method, product, or apparatus that includes said elements is not excluded.

[0022] Although the terms "first," "second," etc., may be used to describe various information in one or more embodiments of this specification, this information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, "first" may also be referred to as "second," and similarly, "second" may also be referred to as "first," without departing from the scope of one or more embodiments of this specification. Ordinal numbers such as "first," "second," etc., do not necessarily indicate order; often they are used to facilitate the distinction of objects. For example, "first server" and "second server" usually refer to two servers. To distinguish these two servers, they are described as "first server" and "second server." Of course, sometimes these two servers may be the same server.

[0023] Depending on the context, the word "if" as used here can be interpreted as "when," "when," or "in response to determination."

[0024] In this specification, unless explicitly stated otherwise, "receiving and sending data" does not necessarily mean direct receiving and sending; it can also mean indirect receiving and sending. For example, A receiving data sent by B can be understood as A directly receiving the data sent by B, or it can be understood as A indirectly receiving the data sent by B through other entities such as C. Similarly, B sending data to A can be understood as B sending the data directly to A, or it can be understood as B indirectly sending the data to A through other entities such as C. Here, C can be one entity, or it can be two or more entities.

[0025] In this specification, unless explicitly stated otherwise, the relationships between structures can be direct or indirect. For example, when describing "A is connected to B," unless it is explicitly stated that A and B are directly connected, it should be understood that A can be directly connected to B or indirectly connected to B. Similarly, when describing "A is on top of B," unless it is explicitly stated that A is directly above B (AB is adjacent and A is above B), it should be understood that A can be directly above B or indirectly above B (AB is separated by other elements, and A is above B). And so on.

[0026] Text-to-speech (TTS), as one of the core technologies of human-computer interaction, aims to convert text into high-quality, natural, and fluent speech, and is widely used in fields such as intelligent assistants, audiobooks, and assistive communication devices. However, existing speech synthesis methods still face significant challenges in generating high-quality speech.

[0027] In practical applications, the autoregressive model (AR) can accurately model the alignment relationship between text and speech by generating acoustic features frame by frame, but its serial generation mechanism leads to slow inference speed.

[0028] Non-autoregressive models (NARs) can significantly accelerate the inference process through parallel generation mechanisms and better maintain global style consistency. However, their fixed-length generation mechanism limits the dynamic adaptation of speech duration and content, leading to problems such as abnormal speech rate or prosodic fragmentation in the generated speech, and the quality of the generated speech still needs to be improved.

[0029] To address the shortcomings of related technologies, this solution provides the following embodiments: Figure 1 This is a schematic diagram illustrating an application scenario of a training method for a speech unit sequence generation model provided in one embodiment of this specification. For example... Figure 1 As shown, this application scenario includes server 1 and speech unit sequence generation model 2 to be trained.

[0030] Server 1 can be an independent physical server, a server cluster consisting of multiple physical servers, or a distributed file system. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms.

[0031] The speech unit sequence generation model 2 to be trained can be a model used to generate speech unit sequences. Optionally, the speech unit sequence generation model 2 to be trained can be a model obtained by pre-training a base model.

[0032] In the embodiments of this specification, server 1 can obtain the sample speech unit sequence corresponding to the sample text, perform a masking operation on the sample speech unit sequence to obtain a first speech unit sequence, perform a length change operation on the first speech unit sequence to obtain a second speech unit sequence, the second speech unit sequence contains functional speech units generated by the length change operation, and then server 1 can train the speech unit sequence generation model 2 to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain the trained speech unit sequence generation model.

[0033] The training method of the speech unit sequence generation model provided in the embodiments of this specification will be described below with reference to the accompanying drawings.

[0034] Figure 2 This is a flowchart illustrating a training method for a speech unit sequence generation model according to an embodiment of this specification. From a programming perspective, the entity executing the process can be a program mounted on a server, model training platform, or terminal. From a hardware perspective, the entity executing the process can be a server, model training platform, or terminal capable of training the model. Figure 2 As shown, the method may include the following steps.

[0035] Step 202: Obtain the sample speech unit sequence corresponding to the sample text.

[0036] In the embodiments of this specification, the sample text and the corresponding sample speech unit sequence can be used as training samples to train the speech unit sequence generation model.

[0037] Optionally, the sample speech unit sequence can be a discrete representation of the speech waveform corresponding to the sample text. The sample speech unit sequence may contain at least one speech unit, which can also be understood as a speech token. In the embodiments of this specification, the speech unit can represent speech semantic information. This speech semantic information can refer to the language content and acoustic information carried by the speech, such as prosody, rhythm, pitch, stress, timbre, etc.

[0038] In the embodiments of this specification, the speech waveform corresponding to the sample text can be obtained first, and then the sample speech unit sequence corresponding to the sample text can be obtained based on the speech waveform. For example, the speech waveform corresponding to the sample text can be obtained by manually reading the sample text aloud, and then the speech waveform can be encoded using an encoder to obtain the sample speech unit sequence corresponding to the sample text.

[0039] As one specific implementation method, the sample text can be downloaded from the Internet, manually written, or generated using a large language model. Optionally, the sample text can be a sentence or a paragraph, such as "The weather is so nice today!" or "The weather is so nice today, but I'm in a bad mood."

[0040] Step 204: Perform a masking operation on the sample speech unit sequence to obtain the first speech unit sequence.

[0041] Masking operations are techniques used in machine learning and deep learning to hide or filter parts of the input data during model training, thereby enhancing the model's generalization ability and robustness. The core of masking operations is to replace parts of a sequence with predefined mask symbols such as [MASK] or specific numerical placeholders, forcing the model to break free from its dependence on the complete sequence. This allows the model to complete prediction tasks by capturing contextual relationships and uncovering inherent patterns in the data, ultimately improving the model's speech reasoning ability, context dependency capture ability, and generalization ability in complex scenarios.

[0042] In the embodiments of this specification, masking a sample speech unit sequence can be performed on the speech units within the sequence. Optionally, one speech unit or multiple speech units within the sequence can be masked.

[0043] Step 206: Perform a length change operation on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation.

[0044] In the embodiments of this specification, the length change operation can be an operation for changing the length of the first speech unit sequence, such as at least one of an operation to increase the length of the first speech unit sequence and an operation to shorten the length of the first speech unit sequence. Optionally, the length of the first speech unit sequence can be determined by the number of speech units in the first speech unit sequence.

[0045] In one specific implementation, the length modification operation can be an operation that increases the length of the first speech unit sequence. Optionally, the length modification operation can be an operation that inserts blank speech units into the first speech unit sequence. The blank speech units can be units without actual speech information.

[0046] In one specific implementation, the length modification operation can be an operation to shorten the length of the first speech unit sequence. Optionally, the length modification operation can be an operation to merge multiple speech units in the first speech unit sequence into one merged speech unit. For example, two speech units in the first speech unit sequence can be merged into one merged speech unit, three speech units in the first speech unit sequence can be merged into one merged speech unit, and so on. Optionally, the second speech unit sequence obtained after performing the operation to shorten the length of the first speech unit sequence may include one merged speech unit or multiple merged speech units.

[0047] In the embodiments of this specification, the second speech unit sequence may include functional speech units generated by a length change operation. If the length of the first speech unit sequence is increased, the functional speech unit may be a blank speech unit. If the length of the first speech unit sequence is shortened, the functional speech unit may be a merged speech unit.

[0048] Step 208: Perform a masking operation on the functional speech units in the second speech unit sequence to obtain the third speech unit sequence.

[0049] In the embodiments described in this specification, the functional voice units in the second voice unit sequence can be masked.

[0050] Optionally, a masking operation can be performed on all functional speech units in the second speech unit sequence. Alternatively, in one specific implementation, a masking operation can be performed on only some of the functional speech units in the second speech unit sequence.

[0051] Step 210: Based on the sample text, the sample speech unit sequence, and the third speech unit sequence, train the speech unit sequence generation model to be trained to obtain the trained speech unit sequence generation model.

[0052] In the embodiments of this specification, the speech unit sequence generation model to be trained can be based on sample text, sample speech unit sequences, and third speech unit sequences. For example, sample text and third speech unit sequences can be input into the speech unit sequence generation model to be trained, and the sample speech unit sequences can be used as prediction targets to train the speech unit sequence generation model.

[0053] Optionally, during the model training process, the to-be-trained speech unit sequence generation model may convert the sample text into its corresponding phoneme unit sequence, so that the to-be-trained speech unit sequence generation model can predict the sample speech unit sequence based on the phoneme unit sequence corresponding to the sample text. For example, input the phoneme unit sequence corresponding to the sample text and the third speech unit sequence into the to-be-trained speech unit sequence generation model, and make the to-be-trained speech unit sequence generation model predict the sample speech unit sequence to complete the model training.

[0054] In the embodiments of this specification, the phoneme unit sequence may be a sequence including the phonemes of the sample text.

[0055] In the text-to-speech synthesis technology scenario, a phoneme refers to the smallest and indivisible basic pronunciation unit that constitutes speech, and is the smallest sound segment that can distinguish semantics in human language. For example, the pronunciation of the Chinese character "欢 (huān)" can be disassembled into three phonemes: / h / , / u / , and / ān / , and each phoneme cannot be further split into smaller pronunciation units with semantic meanings.

[0056] In practical applications, the to-be-trained speech unit sequence generation model may use a grapheme-to-phoneme (G2P) module to convert each text in the sample text into phonemes, so as to obtain the phoneme unit sequence corresponding to the sample text.

[0057] Optionally, the to-be-trained speech unit sequence generation model may be a model for generating speech unit sequences. As a specific implementation manner, the to-be-trained speech unit sequence generation model may be a non-autoregressive model.

[0058] Optionally, during the model training process, the to-be-trained speech unit sequence generation model may predict the functional speech units in the third speech unit sequence, so as to process the functional speech units to adjust the length of the third speech unit sequence. Based on the third speech unit sequence after length adjustment, predict the speech units represented by the masked positions in the third speech unit sequence.

[0059] Optionally, the trained speech unit sequence generation model may be used to predict the speech unit sequence of the text. Further, the predicted speech unit sequence may be decoded to obtain an audio file, and the audio file may be the speech waveform of the text.

[0060] In the embodiments of this specification, the trained speech unit sequence generation model can be applied to various text-to-speech synthesis scenarios. For example, it can be applied to audiobook scenarios, intelligent customer service voice response scenarios, voice navigation scenarios, and so on. For example, in an audiobook scenario, the text of the text to be processed can be obtained first, and then the text can be input into the trained speech unit sequence generation model. The trained speech unit sequence generation model can predict the speech unit sequence corresponding to the text of the text to be processed based on this text. Furthermore, a decoder can be used to decode the predicted speech unit sequence to obtain the audio file of the text to be processed.

[0061] While one or more embodiments of this specification provide method steps as described in the embodiments or flowcharts, it is understood that the order of steps listed in the embodiments or flowcharts is merely one possible execution order among many steps and does not represent the only possible execution order. The order of some steps may be adjusted according to actual needs, or some steps may be omitted. When the claims involve method steps, changes in the order of such steps, or parallel execution between steps, are also within the scope of protection of the claims.

[0062] In practical applications, non-autoregressive models can generate the entire speech unit sequence in parallel through a parallel generation mechanism, without relying on the output of the previous time step to predict the next speech unit as autoregressive models do, thus significantly improving prediction speed. However, existing non-autoregressive models need to pre-predict the fixed total length of the speech unit sequence during the prediction task and allocate phoneme durations proportionally according to this fixed total length to predict the speech unit sequence. If the prediction of the total length of the speech unit sequence is biased, such as being too long or too short, it will cause the speech waveform generated based on the predicted speech unit sequence to have distorted speech rate or prosodic fragmentation, affecting speech quality.

[0063] In the embodiments of this specification, a second speech unit sequence can be obtained by performing a length modification operation on the first speech unit sequence, and a third speech unit sequence can be obtained by masking the functional speech units in the second speech unit sequence. This allows the speech unit sequence generation model to be trained based on sample text, sample speech unit sequences, and the third speech unit sequence. During training, the model can predict the functional speech units in the third speech unit sequence, process these functional speech units to adjust the length of the third speech unit sequence, and then predict the speech units represented by the masked positions in the third speech unit sequence. This enables the trained speech unit sequence generation model to generate speech unit sequences of a length matching the text to be processed, improving the realism and naturalness of the generated speech unit sequences, thereby improving the quality of speech generation.

[0064] based on Figure 2In addition to the method described herein, this specification also provides some implementation methods of the method, which will be described below.

[0065] In the embodiments of this specification, the length modification operation can be an operation to shorten the length of the first speech unit sequence. Optionally, performing the length modification operation on the first speech unit sequence may specifically include: merging at least two adjacent speech units in the first speech unit sequence into a merged speech unit.

[0066] The masking operation on the functional speech units in the second speech unit sequence specifically includes: performing a masking operation on the merged speech units.

[0067] Optionally, at least two adjacent speech units in the first speech unit sequence can be merged into one merged speech unit. For example, two adjacent speech units can be merged into one merged speech unit, or five adjacent speech units can be merged into one merged speech unit.

[0068] In one specific implementation, a group of adjacent speech units in the first speech unit sequence can be merged into a single merged speech unit, thereby obtaining a second speech unit sequence containing one merged speech unit. Alternatively, multiple groups of adjacent speech units in the first speech unit sequence can be merged into multiple merged speech units, thereby obtaining a second speech unit sequence containing multiple merged speech units.

[0069] In the embodiments of this specification, the merged speech units can also be masked to facilitate prediction by the model to be trained, and the predicted merged speech units can be processed, such as expanding the merged speech units into a preset number of mask positions, to increase the length of the speech unit sequence.

[0070] In one specific implementation, the length modification operation can be an operation that increases the length of the first speech unit sequence. Optionally, performing the length modification operation on the first speech unit sequence may specifically include: inserting blank speech units into the first speech unit sequence.

[0071] The masking operation on the functional speech units in the second speech unit sequence specifically includes: masking the blank speech units.

[0072] In the embodiments described in this specification, blank speech units can be inserted before or after any speech unit in the first speech unit sequence. Optionally, one blank speech unit can be inserted into the first speech unit sequence, or multiple blank speech units can be inserted. A blank speech unit can be a unit without actual speech information.

[0073] In one specific implementation, a blank speech unit can be inserted at a certain position in the first speech unit sequence, or multiple blank speech units can be inserted.

[0074] In the embodiments of this specification, blank speech units can also be masked to facilitate prediction by the model to be trained, and the predicted blank speech units can be processed, such as deleting blank speech units from the speech unit sequence to shorten the length of the speech unit sequence.

[0075] Optionally, embodiments of this specification can adjust the parameters of the speech unit sequence generation model to be trained based on the loss value between the sample speech unit sequence and the speech unit sequence predicted by the speech unit sequence generation model to be trained, thereby obtaining the trained speech unit sequence generation model. As a specific implementation, training the speech unit sequence generation model to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain the trained speech unit sequence generation model may specifically include: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained predicts the mask position in the third speech unit sequence based on the sample text, and obtains the first predicted speech unit sequence.

[0076] A first loss value is calculated based on the sample speech unit sequence, the second speech unit sequence, and the first predicted speech unit sequence.

[0077] The parameters of the speech unit sequence generation model to be trained are adjusted based on the first loss value to obtain the trained speech unit sequence generation model.

[0078] In the embodiments of this specification, sample text and a third speech unit sequence can be input into the speech unit sequence generation model to be trained. The speech unit sequence generation model to be trained can predict each mask position in the third speech unit sequence based on the sample text, such as predicting that the mask position is a certain actual speech unit or a certain functional speech unit, to obtain the first predicted speech unit sequence.

[0079] Optionally, a first loss value is calculated based on the sample speech unit sequence, the second speech unit sequence, and the first predicted speech unit sequence. Specifically, this can be achieved by replacing the mask units (excluding functional speech units) in the second speech unit sequence with the corresponding speech units in the sample speech unit sequence to obtain the first target speech unit sequence, thereby calculating the first loss value between the first predicted speech unit sequence and the first target speech unit sequence. Optionally, the first loss value can be calculated using a loss function, such as calculating the mean squared error loss or cross-entropy loss.

[0080] As a specific implementation method, the parameters of the speech unit sequence generation model to be trained can be adjusted according to the first loss value. For example, the parameters of the speech unit sequence generation model to be trained can be adjusted through backpropagation to minimize the difference between the first predicted speech unit sequence and the target speech unit sequence, so as to obtain the trained speech unit sequence generation model.

[0081] Optionally, during training, the model can identify functional speech units in the third speech unit sequence. As a specific implementation, inputting the sample text and the third speech unit sequence into the speech unit sequence generation model to be trained, so that the model predicts the mask positions in the third speech unit sequence based on the sample text, may specifically include: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, and the probability distribution generated by the speech unit sequence generation model to be trained for each mask position in the third speech unit sequence based on the sample text is obtained; the probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; Based on the probability distribution, the functional speech units in each mask position are identified.

[0082] In the embodiments of this specification, sample text and a third speech unit sequence can be input into the speech unit sequence generation model to be trained. The speech unit sequence generation model to be trained can predict the mask positions in the third speech unit sequence based on the sample text, thereby generating a probability distribution for each mask position in the third speech unit sequence. This probability distribution can represent the probability that the mask position is any actual speech unit or any functional speech unit. An actual speech unit can refer to a unit with speech information, and a functional speech unit refers to a unit without speech information. For example, for a certain mask position, the speech unit sequence generation model to be trained predicts that the probability of the mask position being / h / is 90%, the probability of being / e / is 8%, the probability of being a merged speech unit is 1%, and the probability of being a blank speech unit is 1%. Therefore, the probability of / h / being 90%, the probability of / e / being 8%, the probability of being a merged speech unit being 1%, and the probability of being a blank speech unit being 1% can be understood as the probability distribution of that mask position.

[0083] Optionally, functional speech units (SSUs) in each mask location can be identified based on a probability distribution generated by the model. Specifically, the SSUs in each mask location can be identified based on the probability distribution and a first threshold. For example, for any mask location, if the probability that any mask location is a functional speech unit is greater than the first threshold, then that mask location can be identified as a functional speech unit.

[0084] As one specific implementation, after identifying the functional speech units at each mask position in the third speech unit sequence, the model can further process the identified functional speech units to adjust the length of the third speech unit sequence. Optionally, after identifying the functional speech units at each mask position according to the probability distribution, the method may further include: If the functional speech unit is a merged speech unit, then the functional speech unit is expanded into a preset number of mask positions; the merged speech unit is a speech unit formed by merging at least two adjacent speech units in the first speech unit sequence.

[0085] If the functional speech unit is a blank speech unit, then the functional speech unit is deleted; the blank speech unit is a speech unit inserted in the first speech unit sequence.

[0086] Optionally, if the model identifies the mask position as a merged speech unit, the mask position can be expanded to a preset number of mask positions. Here, a merged speech unit is a speech unit formed by merging at least two adjacent speech units in the first speech unit sequence.

[0087] In the embodiments of this specification, expanding a mask position to a preset number of mask positions can mean expanding the mask position to two or more mask positions. Optionally, expanding a mask position to a preset number of mask positions can also mean adding one or more mask positions before or after the current mask position. The training process of the speech unit sequence generation model to be trained in the embodiments of this specification can be a multi-round iterative process. In each round of training, if the model identifies a certain mask position as a merged speech unit, it can expand that mask position to a preset number of mask positions. In the next round of training, the model can predict the probability distribution of these multiple mask positions to identify them.

[0088] Optionally, expanding the mask positions to a preset number of mask positions can increase the length of the third speech unit sequence.

[0089] In one specific implementation, if the model identifies a mask position as a blank speech unit, then that mask position can be deleted, which can also be understood as deleting the blank speech unit. Here, a blank speech unit is a speech unit inserted into the first speech unit sequence. Optionally, deleting the mask position can shorten the length of the third speech unit sequence.

[0090] In the embodiments of this specification, the speech information corresponding to the mask position can also be determined based on the probability distribution of each mask position in the third speech unit sequence. Optionally, the method may further include: Select a mask position that meets the preset filtering conditions from the mask positions outside the functional voice unit.

[0091] Based on the probability distribution of the mask position to be parsed, the voice information corresponding to the mask position to be parsed is determined.

[0092] In the embodiments of this specification, the mask position to be parsed that meets the preset filtering conditions can be a mask position whose probability is greater than or equal to a preset probability threshold in the corresponding probability distribution. This probability greater than or equal to the preset probability threshold can represent the probability that the mask position is any actual speech unit. For example, there are three mask positions: a first mask position, a second mask position, and a third mask position. The first mask position has a 95% probability of being predicted as the actual speech unit / p / and a 5% probability of being predicted as the actual speech unit / b / . The second mask position has an 80% probability of being predicted as the actual speech unit / e / and a 20% probability of being predicted as the actual speech unit / j / . The third mask position has a 40% probability of being predicted as the actual speech unit / n / , a 20% probability of being predicted as the actual speech unit / m / , and a 20% probability of being predicted as the actual speech unit / k / . Assuming the preset probability threshold is 90%, and the probability of the first mask position being predicted as the actual speech unit / p / is greater than the probability threshold of 90%, then the first mask position can be determined as the mask position to be parsed. Optionally, the preset probability threshold can be a value above 50%, such as 80% or 90%, etc., without specific limitations.

[0093] In one specific implementation, the mask position to be parsed that meets the preset filtering conditions can be the mask position corresponding to the first probability after arranging the probabilities of each mask position as each actual speech unit in order of magnitude. The probabilities of each mask position as each actual speech unit include a first probability and a second probability, with the first probability being greater than the second probability.

[0094] Optionally, the mask positions to be parsed that meet the preset filtering conditions can be a preset number or a preset proportion of mask positions determined from mask positions other than the functional voice units.

[0095] Optionally, the mask position to be parsed that meets the preset filtering conditions can be a single mask position or multiple mask positions.

[0096] In the embodiments of this specification, the speech information corresponding to the position of the mask to be parsed can be determined based on the probability distribution of the position of the mask to be parsed. For example, the actual speech unit with the highest probability can be determined from the probability distribution of the position of the mask to be parsed, and thus the actual speech unit can be determined as the speech information corresponding to the position of the mask to be parsed.

[0097] In practical applications, to enable the model to generate speech unit sequences with different timbre information, thereby obtaining speech waveforms with different timbre information and improving user experience, the embodiments of this specification can utilize reference timbre information to train the speech unit sequence generation model to be trained. Optionally, the method may further include: Obtain reference timbre information.

[0098] The step of training the speech unit sequence generation model based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain the trained speech unit sequence generation model may specifically include: The sample text and the reference timbre information are input into the speech unit sequence generation model to be trained, and the second predicted speech unit sequence output by the speech unit sequence generation model to be trained is obtained.

[0099] The second loss value is calculated based on the sample speech unit sequence, the second speech unit sequence, and the second predicted speech unit sequence.

[0100] The parameters of the speech unit sequence generation model to be trained are adjusted based on the second loss value to obtain the trained speech unit sequence generation model.

[0101] Timbre, also known as tone quality or tone color, is one of the subjective acoustic attributes of sound. It refers to the essential sound characteristics that can distinguish different sound sources or methods of sound production, provided that the pitch and loudness are the same.

[0102] In the embodiments of this specification, the reference timbre information may include various timbre information, such as timbre information representing a person's joy, anger, sadness, youth, or old age, etc. Optionally, the reference timbre information can be used as training samples to train the model, enabling it to generate various timbre information.

[0103] In practical applications, obtaining reference timbre information can specifically involve obtaining a discrete representation of a reference speech waveform. This discrete representation of the reference speech waveform can carry timbre reference information used to guide the model in generating speech unit sequences, as well as reference information such as prosody, rhythm, pitch, and stress.

[0104] Optionally, sample text and reference timbre information can be input into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained outputs a second predicted speech unit sequence.

[0105] In the embodiments of this specification, a second loss value can be calculated based on the sample speech unit sequence, the second speech unit sequence, and the second predicted speech unit sequence. Optionally, the mask units in the second speech unit sequence, excluding functional speech units, can be replaced with the corresponding speech units in the sample speech unit sequence to obtain the target speech unit sequence, thereby allowing the calculation of the second loss value between the second predicted speech unit sequence and the target speech unit sequence.

[0106] As a specific implementation method, the parameters of the speech unit sequence generation model to be trained can be adjusted according to the second loss value. For example, the parameters of the speech unit sequence generation model to be trained can be adjusted through backpropagation to minimize the difference between the second predicted speech unit sequence and the target speech unit sequence, so as to obtain the trained speech unit sequence generation model.

[0107] Figure 3 This is a schematic diagram illustrating a scenario for acquiring a third speech unit sequence according to an embodiment of this specification. For example... Figure 3 As shown, the embodiments of this specification can perform a masking operation on the sample speech unit sequence S0 corresponding to the sample text, where the masking operation can also be understood as a noise addition operation. Masking the sample speech unit sequence S0 yields a first speech unit sequence S1 containing the mask. Further, the embodiments of this specification can perform a length modification operation on the first speech unit sequence S1, such as merging at least two adjacent speech units in the first speech unit sequence S1 into a merged speech unit, or inserting blank speech units into the first speech unit sequence S1, thereby obtaining a second speech unit sequence S2. Then, functional speech units in the second speech unit sequence S2, such as merged speech units and blank speech units, can be masked to obtain a third speech unit sequence S3.

[0108] Figure 4 This is a schematic diagram illustrating a scenario for training a speech unit sequence generation model, as provided in one embodiment of this specification. For example... Figure 4As shown, the phoneme unit sequence corresponding to the sample text and the third speech unit sequence S3 can be input into the speech unit sequence generation model to be trained. As a specific implementation, reference timbre information can also be input into the speech unit sequence generation model to be trained. The speech unit sequence generation model to be trained can predict the masks in the third speech unit sequence S3, generating probability distributions for each mask position. Further, the speech unit sequence generation model to be trained can identify functional speech units in the third speech unit sequence S3 based on the probability distributions. Simultaneously, the speech unit sequence generation model to be trained can also select mask positions that meet preset filtering conditions from mask positions outside the functional speech units in the third speech unit sequence S3, and then determine the speech information corresponding to the mask positions to be parsed based on the probability distributions. Thus, a speech unit sequence S4 with identified functional speech units and determined speech information corresponding to the mask positions to be parsed can be obtained. In the embodiments of this specification, the speech unit sequence generation model to be trained can also process the functional speech units to adjust the length of the third speech unit sequence S3. For example, the merged speech units can be expanded to 5 mask positions, and blank speech units can be deleted from the third speech unit sequence S3 to obtain the speech unit sequence S5 with adjusted length. The length of the third speech unit sequence S3 can be equal to the length of the speech unit sequence S4.

[0109] Furthermore, continuing the above approach, the speech unit sequence generation model to be trained can be instructed to predict the masks in the speech unit sequence S5 based on the sample text until all masks are predicted, thereby obtaining the first predicted speech unit sequence.

[0110] Optionally, a first loss value can be calculated between the first predicted speech unit sequence and the sample speech unit sequence, and the parameters of the speech unit sequence generation model to be trained can be adjusted based on the first loss value to obtain the trained speech unit sequence generation model.

[0111] Based on the same idea, this specification also provides a method for generating speech unit sequences. Figure 5 This is a flowchart of a speech unit sequence generation method provided in one embodiment of this specification, such as... Figure 5 As shown, the speech unit sequence generation method may include the following steps.

[0112] Step 502: Obtain the text to be processed.

[0113] In the embodiments described in this specification, the text to be processed can be the text that needs to be generated into speech.

[0114] Step 504: Input the text to be processed into the trained speech unit sequence generation model so that the trained speech unit sequence generation model generates a first probability distribution of each mask position in the preset speech unit sequence; the first probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit.

[0115] Optionally, after inputting the text to be processed into the trained speech unit sequence generation model, the trained speech unit sequence generation model can convert the text to be processed into a phoneme sequence. For example, the trained speech unit sequence generation model can use a character-to-phoneme module to convert the text to be processed into individual phoneme units, thereby obtaining the phoneme sequence corresponding to the text to be processed.

[0116] In the embodiments of this specification, after inputting the text to be processed into the trained speech unit sequence generation model, the trained speech unit sequence generation model can generate a first probability distribution of each mask position in the preset speech unit sequence. The preset speech unit sequence can be an object template used for speech unit sequence prediction. The preset speech unit sequence can be a sequence of all masks. The trained speech unit sequence generation model can predict the actual speech unit or functional speech unit at each mask position in the preset speech unit sequence based on the text to be processed, such as the phoneme sequence corresponding to the text to be processed, thereby generating a predicted speech unit sequence corresponding to the phoneme sequence to be processed. Optionally, the preset speech unit sequence can be a sequence of arbitrary length, such as a sequence containing one mask position or a sequence containing multiple mask positions.

[0117] In practical applications, any mask position in a preset speech unit sequence can represent an actual speech unit or a functional speech unit. In the embodiments of this specification, the trained speech unit sequence generation model can generate a first probability distribution for each mask position in the preset speech unit sequence. This first probability distribution can represent the probability that the corresponding mask position is any actual speech unit or any functional speech unit. For example, for a certain mask position, the first probability distribution generated by the trained speech unit sequence generation model could be that the probability of that mask position being the actual speech unit / p / is 95%, the probability of being the actual speech unit / b / is 3%, the probability of being a merged speech unit is 1%, and the probability of being a blank speech unit is 1%.

[0118] Optionally, the trained speech unit sequence generation model can be obtained by training the model using the training method of the previous speech unit sequence generation model.

[0119] Step 506: The trained speech unit sequence generation model identifies the functional speech units at each mask position in the preset speech unit sequence according to the first probability distribution.

[0120] In the embodiments of this specification, the trained speech unit sequence generation model can identify functional speech units at each mask position in a preset speech unit sequence based on a first probability distribution. For example, for a certain mask position, if the probability of that mask position being predicted as a functional speech unit is greater than a preset threshold, then that mask position can be determined as a functional speech unit.

[0121] Step 508: Perform length change inverse processing on the functional speech unit to obtain the first processed speech unit sequence.

[0122] In the embodiments of this specification, the inverse length modification processing of functional voice units can be performed on the length of the functional voice unit. For example, if the functional voice unit is a merged voice unit, it can be expanded to a preset number of mask positions. If the functional voice unit is a blank voice unit, it can be deleted from the preset voice unit sequence.

[0123] Optionally, performing inverse length alteration processing on the functional speech units can involve adjusting the length of a preset speech unit sequence. As one specific implementation, the length of the first processed speech unit sequence obtained by performing inverse length alteration processing on the functional speech units can be the same as or different from the length of the preset speech unit sequence.

[0124] Step 510: Based on the first probability distribution, determine the speech information corresponding to the mask position that meets the preset filtering conditions in the first processed speech unit sequence, and obtain the second processed speech unit sequence.

[0125] In the embodiments of this specification, mask positions in the first processed speech unit sequence that meet preset filtering conditions can be determined first. These mask positions that meet the preset filtering conditions can be mask positions whose probability of being predicted as a certain actual speech unit is greater than a certain threshold. For example, one mask position may have a 95% probability of being predicted as the actual speech unit / p / and a 5% probability of being predicted as the actual speech unit / b / . Another mask position may have an 80% probability of being predicted as the actual speech unit / e / and a 20% probability of being predicted as the actual speech unit / j / . If the threshold is set to 90%, then a mask position with a 95% probability of being predicted as the actual speech unit / p / can be a mask position that meets the preset filtering conditions.

[0126] Optionally, the mask positions that meet the preset filtering conditions can be the mask positions corresponding to the first probability after arranging the probabilities of each mask position as each actual speech unit in order of magnitude. The probabilities of each mask position as each actual speech unit include a first probability and a second probability, with the first probability being greater than the second probability.

[0127] In one specific implementation, the speech information corresponding to the mask positions that meet the preset filtering conditions can be determined based on the first probability distribution. For example, the actual speech unit with the highest predicted probability can be used as the speech information corresponding to the mask positions that meet the preset filtering conditions, thereby obtaining the second processed speech unit sequence.

[0128] In this embodiment, the execution order of steps 506, 508, and 510 is not limited. For example, step 506 can be executed first, where the trained speech unit sequence generation model identifies functional speech units at each mask position in the preset speech unit sequence according to the first probability distribution. Then, step 508 is executed, performing inverse length modification processing on the functional speech units in the preset speech unit sequence. Afterward, step 510 is executed to determine the speech information corresponding to the mask positions that meet the preset filtering conditions. As a specific implementation, step 510 can also be executed first. When step 510 is executed first, it can be that the speech information corresponding to the mask positions that meet the preset filtering conditions in the preset speech unit sequence is determined according to the first probability distribution. Then, step 506 is executed, where the trained speech unit sequence generation model identifies functional speech units at each mask position in the preset speech unit sequence according to the first probability distribution. Afterward, step 508 is executed, performing inverse length modification processing on the functional speech units in the preset speech unit sequence.

[0129] Step 512: Predict the mask positions in the second processed speech unit sequence to obtain the predicted speech unit sequence.

[0130] In practical applications, the trained speech unit sequence generation model can identify functional speech units at various mask positions in the preset speech unit sequence based on the first probability distribution. These functional speech units may be merged speech units. Therefore, performing an inverse length-changing process on these functional speech units may expand the preset speech unit sequence by a preset number of mask positions. During the prediction of mask positions in the second-processed speech unit sequence, predictions can be made targeting these mask positions.

[0131] Optionally, the first processed speech unit sequence may contain multiple mask positions, and the mask positions that meet the preset filtering conditions may be some of these mask positions, while the remaining mask positions may be mask positions that do not meet the preset filtering conditions. During the prediction of mask positions in the second processed speech unit sequence, prediction can be performed on these mask positions that do not meet the preset filtering conditions.

[0132] In the embodiments of this specification, the trained speech unit sequence generation model can predict the probability that each mask position in a preset speech unit sequence is any actual speech unit or any functional speech unit. It can then perform inverse length modification processing on the functional speech units to adjust the length of the preset speech unit sequence, and subsequently predict the actual speech units at each mask position in the preset speech unit sequence. This enables the trained speech unit sequence generation model to generate speech unit sequences of a length matching the text to be processed, improving the realism and naturalness of the generated speech unit sequences, thereby improving the quality of speech generation.

[0133] As one specific implementation, the functional speech unit includes at least one of a merged speech unit and a blank speech unit. The step of performing a length-changing inverse process on the functional speech unit to obtain a first processed speech unit sequence may specifically include: If the functional voice unit is the merged voice unit, then the functional voice unit is expanded to a preset number of mask positions.

[0134] If the functional voice unit is a blank voice unit, then the functional voice unit is deleted.

[0135] Optionally, the functional speech unit can be a unit for adjusting the length of a preset speech unit sequence. In one specific implementation, the functional speech unit may include at least one of a merged speech unit and a blank speech unit. The merged speech unit may be a unit including multiple mask positions, or a unit including multiple speech units. The blank speech unit may be a unit without actual speech information.

[0136] In the embodiments of this specification, if the functional speech unit is a merged speech unit, the functional speech unit can be expanded to a preset number of mask positions to increase the length of the preset speech unit sequence. For example, the functional speech unit can be replaced with a preset number of mask positions. The specific value of the preset number can be determined according to actual needs, such as 1, 3, 5, etc.

[0137] Optionally, if a functional speech unit is a blank speech unit, it can be deleted from the preset speech unit sequence to shorten the length of the preset speech unit sequence.

[0138] In this embodiment of the specification, the speech information corresponding to the mask position can also be determined based on the probability distribution of each mask position in the preset speech unit sequence. Optionally, determining the speech information corresponding to the mask position that meets the preset filtering conditions in the first processed speech unit sequence based on the first probability distribution to obtain the second processed speech unit sequence may specifically include: From the first processed speech unit sequence, select the mask position that meets the preset filtering conditions.

[0139] Based on the probability distribution of the mask position to be parsed, the speech information corresponding to the mask position to be parsed is determined, and the second processed speech unit sequence is obtained.

[0140] In the embodiments of this specification, the mask position to be parsed that meets the preset filtering conditions can be a mask position with a probability greater than or equal to a preset probability threshold. This probability can represent the probability that the mask position is any actual speech unit. As a specific implementation, the mask position to be parsed that meets the preset filtering conditions can be the mask position corresponding to the first probability after arranging the probabilities of each mask position as each actual speech unit in ascending order. The probabilities of each mask position as each actual speech unit include a first probability and a second probability, where the first probability is greater than the second probability.

[0141] Optionally, the mask position to be parsed that meets the preset filtering conditions can be a single mask position or multiple mask positions.

[0142] In the embodiments of this specification, the speech information corresponding to the position of the mask to be parsed can be determined based on the probability distribution of the position of the mask to be parsed. For example, the actual speech unit with the highest probability can be determined from the probability distribution of the position of the mask to be parsed, and thus the actual speech unit can be determined as the speech information corresponding to the position of the mask to be parsed, so as to obtain the second processed speech unit sequence.

[0143] In practical applications, the second processed speech unit sequence may contain mask positions. After training, the speech unit sequence generation model needs to predict the mask positions in the second processed speech unit sequence to obtain the predicted speech unit sequence. Optionally, predicting the mask positions in the second processed speech unit sequence to obtain the predicted speech unit sequence may specifically include: The trained speech unit sequence generation model predicts the mask positions in the second processed speech unit sequence to obtain a second probability distribution of each mask position in the second processed speech unit sequence.

[0144] Based on the second probability distribution, identify the functional speech units at each mask position in the second processed speech unit sequence.

[0145] The functional speech units at each mask position in the second processed speech unit sequence are subjected to length-changing inverse processing to obtain the third processed speech unit sequence.

[0146] Based on the second probability distribution, the voice information corresponding to the mask position that meets the preset filtering conditions in the third processed voice unit sequence is determined, and the fourth processed voice unit sequence is obtained.

[0147] The mask positions in the fourth processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

[0148] In the embodiments of this specification, the process of the trained speech unit sequence generation model generating a predicted speech unit sequence can be a multi-round prediction process. Each round of prediction can predict some mask positions in the current preset speech unit sequence, and the final predicted speech unit sequence is obtained after multiple rounds of prediction.

[0149] Optionally, the trained speech unit sequence generation model can predict the mask positions in the second processed speech unit sequence to obtain the second probability distribution of each mask position in the second processed speech unit sequence. Then, based on the second probability, it can identify the functional speech units in each mask position in the second processed speech unit sequence. The identified functional speech units are subjected to length change inverse processing to obtain the third processed speech unit sequence. After that, the speech information corresponding to the mask positions in the third processed speech unit sequence that meet the preset screening conditions can be determined to obtain the fourth processed speech unit sequence. The mask positions in the fourth processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

[0150] In the embodiments of this specification, a predicted speech unit sequence corresponding to the prompt timbre information can also be generated based on the prompt timbre information. Optionally, the method may further include: Get the prompt sound information.

[0151] The step of inputting the text to be processed into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence based on the text to be processed, specifically includes: The prompt timbre information and the text to be processed are input into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence based on the prompt timbre information and the text to be processed.

[0152] In the embodiments of this specification, the prompt timbre information can be used to generate a speech unit sequence with the timbre represented by the prompt timbre information after prompt training of the speech unit sequence generation model.

[0153] Optionally, the prompt tone information and the text to be processed can be input into the trained speech unit sequence generation model. The trained speech unit sequence generation model can generate a first probability distribution for each mask position in the preset speech unit sequence based on the prompt tone information and the text to be processed. Then, the trained speech unit sequence generation model can generate the speech unit sequence with the tone represented by the prompt tone information based on the first probability distribution.

[0154] Based on the same idea, this specification also provides a speech synthesis method in its embodiments. Figure 6 This is a flowchart illustrating a speech synthesis method provided in one embodiment of this specification. Figure 6 As shown, the speech synthesis method may include the following steps.

[0155] Step 602: Obtain the text to be processed.

[0156] In the embodiments described in this specification, the text to be processed can be text used for speech synthesis.

[0157] Optionally, the text to be processed can be text from various application scenarios. For example, the text to be processed can be text from a reading material, text used in intelligent customer voice response scenarios, text used in voice navigation scenarios, etc.

[0158] In practical applications, users can send the text to be processed to the execution entity of the process, such as a server, via a terminal, so that the server can obtain the text to be processed.

[0159] Step 604: Input the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the training method of the speech unit sequence generation model mentioned above.

[0160] Optionally, to facilitate the generation of predicted speech unit sequences by the trained speech unit sequence generation model, thereby facilitating the acquisition of the speech waveform of the text to be processed, the trained speech unit sequence generation model in this embodiment can convert the text to be processed into a phoneme unit sequence. Optionally, a character-to-phoneme module can be used to convert the text to be processed into a phoneme unit sequence.

[0161] In the embodiments of this specification, a phoneme unit sequence can be input into a trained speech unit sequence generation model, so that the trained speech unit sequence generation model can generate a predicted speech unit sequence based on the phoneme unit sequence.

[0162] Optionally, the trained speech unit sequence generation model is trained using the training method for the speech unit sequence generation model mentioned above. The specific training process can be found above and will not be repeated here.

[0163] Step 606: Decode the predicted speech unit sequence to obtain the audio file of the text to be processed.

[0164] In the embodiments of this specification, the predicted speech unit sequence can be decoded. For example, the predicted speech unit sequence can be sequentially input into an acoustic model (Acoustic DLM), a decoder, and a vocoder to finally obtain an audio file of the text to be processed. The audio file of the text to be processed can be the speech waveform of the text to be processed.

[0165] As one specific implementation method, the speech synthesis method may further include: Get the prompt sound information.

[0166] The step of inputting the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model may specifically include: The text to be processed and the prompt timbre information are input into the trained speech unit sequence generation model to obtain the predicted speech unit sequence corresponding to the text to be processed, generated by the trained speech unit sequence generation model based on the prompt timbre information.

[0167] In the embodiments of this specification, the prompt timbre information can be used to generate a speech unit sequence with the timbre represented by the prompt timbre information after prompt training of the speech unit sequence generation model.

[0168] Optionally, the text to be processed and the prompt timbre information can be input into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model can generate a speech unit sequence with the timbre represented by the prompt timbre information.

[0169] In related technologies, non-autoregressive models can be used to complete speech generation tasks. However, current non-autoregressive models rely on global duration prediction as initial input, requiring pre-estimation of the total speech duration and proportional allocation of phoneme durations. If the prediction bias is too long or too short, it will lead to distorted speech rate or prosodic fragmentation in the generated speech. Secondly, the local nature of masking operations makes it difficult for these models to capture long-distance dependencies, such as semantic pauses in complex sentences or rhythmic changes required for emotional expression, thus limiting the naturalness of the generated speech. In addition, the masking strategies of these models, such as random masking strategies and dynamic attention strategies, are prone to failure in noisy environments or with unseen corpora, and prediction errors may accumulate, affecting the generation quality. For example, in low-resource language scenarios, prediction bias in masking positions may lead to an imbalance in phoneme distribution, further exacerbating speech incoherence.

[0170] The embodiments in this specification can eliminate the dependence on external duration predictors, enabling the model to dynamically and automatically adjust the length of the speech unit sequence during inference, ultimately achieving high-quality, high-naturalness, and high-flexibility speech synthesis.

[0171] Specifically, the embodiments in this specification introduce a length-changing operation, transforming the control of the speech unit sequence length from "pre-prediction" to "dynamic adjustment on demand." Instead of passively accepting a fixed speech duration before generating the speech unit sequence—that is, passively accepting the length of a speech unit sequence—the model autonomously decides when and where to lengthen or shorten pronunciations based on its understanding of speech semantics during the prediction task. This eliminates the strong dependence on precise duration prediction. This allows the model to ensure the generation of dynamic and varied speech rhythms regardless of the given preset speech unit sequence length, thus more realistically simulating the natural prosody of human speech.

[0172] This specification also proposes a hierarchical probability-guided inference strategy in its embodiments. During the prediction task—that is, during the denoising of a preset speech unit sequence to obtain a predicted speech unit sequence—the model can first perform a unified prediction on each mask position of the preset speech unit sequence to obtain a probability distribution. Then, the model can check the probability of functional speech units, which can also represent a confidence score. Functional speech units with confidence scores exceeding a preset threshold, such as 0.8, can be sampled and retained for subsequent processing. Simultaneously, the model can perform top-k sampling on a portion of the remaining mask positions to determine the actual speech information of those remaining mask positions. Afterward, the model can replace merged speech units in the functional speech units with a preset number of mask positions and delete blank speech units from the speech unit sequence, thereby adjusting the length of the speech unit sequence. This sequence, after content prediction and length adjustment, can be used as input to the model for the next round of denoising. Through multiple rounds of denoising, the model can generate realistic and natural speech.

[0173] The various technical features in the above embodiments can be combined arbitrarily, as long as there is no conflict or contradiction between the combinations of features. However, due to space limitations, they have not been described one by one. Therefore, the arbitrary combination of various technical features in the above embodiments is also within the scope of this specification.

[0174] Based on the same idea, this specification also provides an apparatus corresponding to the training method of the above-mentioned speech unit sequence generation model.

[0175] Figure 7 This is a schematic diagram of the structure of a training device for a speech unit sequence generation model provided in one embodiment of this specification, as shown below. Figure 7As shown, the training device for the speech unit sequence generation model may include: The first acquisition module 702 is used to acquire the sample speech unit sequence corresponding to the sample text.

[0176] The first operation module 704 is used to perform a masking operation on the sample speech unit sequence to obtain the first speech unit sequence.

[0177] The second operation module 706 is used to perform a length change operation on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation.

[0178] The third operation module 708 is used to perform a masking operation on the functional speech units in the second speech unit sequence to obtain the third speech unit sequence.

[0179] The training module 710 is used to train the speech unit sequence generation model to be trained based on the phoneme unit sequence corresponding to the sample text, the sample speech unit sequence, and the third speech unit sequence, so as to obtain the trained speech unit sequence generation model.

[0180] based on Figure 7 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0181] Optionally, the second operation module 706 can be used to: merge at least two adjacent speech units in the first speech unit sequence into a merged speech unit.

[0182] The third operation module 708 can be specifically used to perform a masking operation on the merged speech unit.

[0183] Optionally, the second operation module 706 can be specifically used to: insert blank speech units into the first speech unit sequence; The third operation module 708 can be specifically used to: perform a masking operation on the blank voice unit.

[0184] Optional, training module 710, which can be used for: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained predicts the mask position in the third speech unit sequence based on the sample text, and obtains the first predicted speech unit sequence.

[0185] Calculate the first loss value between the first predicted speech unit sequence and the sample speech unit sequence.

[0186] The parameters of the speech unit sequence generation model to be trained are adjusted based on the first loss value to obtain the trained speech unit sequence generation model.

[0187] Optionally, inputting the sample text and the third speech unit sequence into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained predicts the mask position in the third speech unit sequence based on the sample text, may specifically include: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, and the probability distribution of each mask position generated by the speech unit sequence generation model to be trained based on the sample text is obtained. The probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit.

[0188] Based on the probability distribution, the functional speech units in each mask position are identified.

[0189] Optionally, the device may further include: An extension module is used to extend the functional voice unit into a preset number of mask positions if the functional voice unit is a merged voice unit; the merged voice unit is a voice unit formed by merging at least two adjacent voice units in the first voice unit sequence.

[0190] The deletion module is used to delete the functional voice unit if the functional voice unit is a blank voice unit; the blank voice unit is a voice unit inserted in the first voice unit sequence.

[0191] Optionally, the device may further include: The selection module is used to select a mask position that meets preset filtering conditions from the mask positions outside the functional voice unit.

[0192] The voice information determination module is used to determine the voice information corresponding to the position of the mask to be parsed based on the probability distribution of the position of the mask to be parsed.

[0193] Optionally, the device may further include: The reference timbre information acquisition module is used to acquire reference timbre information.

[0194] Training module 710 can be used for: The sample text and the reference timbre information are input into the speech unit sequence generation model to be trained, and the second predicted speech unit sequence output by the speech unit sequence generation model to be trained is obtained.

[0195] Calculate a second loss value between the second predicted speech unit sequence and the sample speech unit sequence.

[0196] The parameters of the speech unit sequence generation model to be trained are adjusted based on the second loss value to obtain the trained speech unit sequence generation model.

[0197] Based on the same idea, this specification also provides an apparatus corresponding to the above-described speech unit sequence generation method.

[0198] Figure 8 This is a schematic diagram of the structure of a speech unit sequence generation device provided in one embodiment of this specification, as shown below. Figure 8 As shown, the speech unit sequence generation device may include: The second acquisition module 802 is used to acquire the text to be processed.

[0199] The first input module 804 is used to input the text to be processed into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution of each mask position in the preset speech unit sequence; the first probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit.

[0200] The recognition module 806 is used to identify the functional speech units in each mask position of the preset speech unit sequence according to the first probability distribution by the trained speech unit sequence generation model.

[0201] The inverse processing module 808 is used to perform length change inverse processing on the functional speech unit to obtain the first processed speech unit sequence.

[0202] The voice information determination module 810 is used to determine the voice information corresponding to the mask position that meets the preset filtering conditions in the first processed voice unit sequence according to the first probability distribution, so as to obtain the second processed voice unit sequence.

[0203] The prediction module 812 is used to predict the mask position in the second processed speech unit sequence to obtain the predicted speech unit sequence.

[0204] based on Figure 8 The embodiments of this specification also provide some specific implementation schemes of the method, which are described below.

[0205] Optionally, the functional speech unit includes at least one of a merged speech unit and a blank speech unit. The inverse processing module 808 can specifically be used for: If the functional voice unit is the merged voice unit, then the functional voice unit is expanded to a preset number of mask positions; If the functional voice unit is a blank voice unit, then the functional voice unit is deleted.

[0206] Optionally, the voice information determination module 810 can be used for: From the first processed speech unit sequence, select the mask position that meets the preset filtering conditions.

[0207] Based on the probability distribution of the mask position to be parsed, the speech information corresponding to the mask position to be parsed is determined, and the second processed speech unit sequence is obtained.

[0208] Optionally, the prediction module 812 can be used for: The trained speech unit sequence generation model predicts the mask positions in the second processed speech unit sequence to obtain a second probability distribution of each mask position in the second processed speech unit sequence.

[0209] Based on the second probability distribution, identify the functional speech units at each mask position in the second processed speech unit sequence.

[0210] The functional speech units at each mask position in the second processed speech unit sequence are subjected to length-changing inverse processing to obtain the third processed speech unit sequence.

[0211] Based on the second probability distribution, the voice information corresponding to the mask position that meets the preset filtering conditions in the third processed voice unit sequence is determined, and the fourth processed voice unit sequence is obtained.

[0212] The mask positions in the fourth processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

[0213] Optionally, the device may further include: The prompt tone information acquisition module is used to acquire prompt tone information.

[0214] The first input module 804 can be used specifically for: The prompt timbre information and the text to be processed are input into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence based on the prompt timbre information and the text to be processed.

[0215] Based on the same idea, this specification also provides an apparatus corresponding to the above-described speech synthesis method.

[0216] Figure 9This is a schematic diagram of the structure of a speech synthesis device provided in one embodiment of this specification, as shown below. Figure 9 As shown, the speech synthesis device may specifically include: The third acquisition module 902 is used to acquire the text to be processed.

[0217] The second input module 904 is used to input the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the training method of the above-mentioned speech unit sequence generation model.

[0218] The decoding module 906 is used to decode the predicted speech unit sequence to obtain the audio file of the text to be processed.

[0219] Based on the same idea, this specification also provides devices corresponding to the above methods in its embodiments.

[0220] Figure 10 A structural block diagram of a computing device 1000 provided according to an embodiment of this specification is shown.

[0221] The computing device 1000 includes: Memory 1010 and processor 1020; The memory 1010 is used to store computer programs / instructions, and the processor 1020 is used to execute the computer programs / instructions, which implement the steps of the above method when executed by the processor 1020.

[0222] Specifically, the components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and the database 1050 is used to store data.

[0223] The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., a network interface card (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Wi-MAX (Worldwide Interoperability for Microwave Access) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, a Near Field Communication (NFC) interface, and so on.

[0224] In one embodiment of this specification, the above-described components of the computing device 1000 and Figure 10 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 10 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this application. Those skilled in the art can add or replace other components as needed.

[0225] The computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1000 can also be a mobile or stationary server.

[0226] The processor 1020 executes the computer instructions to implement the steps of the multimodal data processing method and the multimodal large model training method.

[0227] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device and the technical solution of the steps of the above method belong to the same concept. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solution of the above method.

[0228] An embodiment of this specification also provides a computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the above-described method.

[0229] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the storage medium can be referred to the description of the technical solution of the above method.

[0230] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0231] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product and the technical solution of the above method belong to the same concept, and all details not described in detail in the technical solution of the computer program product can be referred to in the description of the technical solution of the above method.

[0232] The various embodiments in this specification are described in a progressive manner, and the same or similar parts between the embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for the apparatus and device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the description of the method embodiments. The apparatus, device and method provided in the embodiments of this specification are corresponding to each other, and therefore the apparatus and device also have similar beneficial technical effects as the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the corresponding apparatus and device will not be repeated here.

[0233] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0234] In the 1990s, improvements to a technology could be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to methodology). However, with technological advancements, many methodological improvements today can be considered direct improvements to hardware circuit structures. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that a methodological improvement cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (such as a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program a digital system themselves to "integrate" it onto a PLD, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must also be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, the most commonly used are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0235] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0236] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0237] For ease of description, the above devices are described separately by function as various units. Of course, in implementing this application, the functions of each unit can be implemented in one or more software and / or hardware.

[0238] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, the invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0239] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0240] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0241] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0242] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0243] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0244] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital character versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0245] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0246] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.

Claims

1. A training method for a speech unit sequence generation model, comprising: Obtain the sample speech unit sequence corresponding to the sample text; The sample speech unit sequence is masked to obtain the first speech unit sequence; A length change operation is performed on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation. The functional speech units in the second speech unit sequence are masked to obtain the third speech unit sequence; Based on the sample text, the sample speech unit sequence, and the third speech unit sequence, the speech unit sequence generation model to be trained is trained to obtain the trained speech unit sequence generation model.

2. The method as described in claim 1, wherein performing a length modification operation on the first speech unit sequence specifically includes: At least two adjacent speech units in the first speech unit sequence are merged into one merged speech unit; The masking operation on the functional speech units in the second speech unit sequence specifically includes: performing a masking operation on the merged speech units.

3. The method as described in claim 1, wherein performing a length modification operation on the first speech unit sequence specifically includes: Insert blank speech units into the first speech unit sequence; The masking operation on the functional speech units in the second speech unit sequence specifically includes: masking the blank speech units.

4. The method as described in claim 1, wherein training the speech unit sequence generation model to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain the trained speech unit sequence generation model specifically includes: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained can predict the mask position in the third speech unit sequence based on the sample text, and obtain the first predicted speech unit sequence. Calculate a first loss value based on the sample speech unit sequence, the second speech unit sequence, and the first predicted speech unit sequence; The parameters of the speech unit sequence generation model to be trained are adjusted based on the first loss value to obtain the trained speech unit sequence generation model.

5. The method as described in claim 4, wherein inputting the sample text and the third speech unit sequence into the speech unit sequence generation model to be trained, so that the speech unit sequence generation model to be trained predicts the mask position in the third speech unit sequence based on the sample text, specifically includes: The sample text and the third speech unit sequence are input into the speech unit sequence generation model to be trained, and the probability distribution generated by the speech unit sequence generation model to be trained for each mask position in the third speech unit sequence based on the sample text is obtained; the probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; Based on the probability distribution, the functional speech units in each mask position are identified.

6. The method of claim 5, further comprising, after identifying the functional speech units at each mask location according to the probability distribution: If the functional voice unit is a merged voice unit, then the functional voice unit is expanded into a preset number of mask positions; The merged speech unit is a speech unit formed by merging at least two adjacent speech units in the first speech unit sequence; If the functional speech unit is a blank speech unit, then the functional speech unit is deleted; the blank speech unit is a speech unit inserted in the first speech unit sequence.

7. The method of claim 5, further comprising: Select a mask position that meets the preset filtering conditions from the mask positions outside the functional voice unit; Based on the probability distribution of the mask position to be parsed, the voice information corresponding to the mask position to be parsed is determined.

8. The method of claim 1, further comprising: Obtain reference timbre information; The step of training the speech unit sequence generation model based on the sample text, the sample speech unit sequence, and the third speech unit sequence to obtain the trained speech unit sequence generation model specifically includes: The sample text and the reference timbre information are input into the speech unit sequence generation model to be trained, and the second predicted speech unit sequence output by the speech unit sequence generation model to be trained is obtained. Calculate the second loss value based on the sample speech unit sequence, the second speech unit sequence, and the second predicted speech unit sequence; The parameters of the speech unit sequence generation model to be trained are adjusted based on the second loss value to obtain the trained speech unit sequence generation model.

9. A method for generating a speech unit sequence, comprising: Get the text to be processed; The text to be processed is input into the trained speech unit sequence generation model so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence; the first probability distribution for each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; The trained speech unit sequence generation model identifies functional speech units at each mask position in the preset speech unit sequence based on the first probability distribution. The functional speech unit is subjected to a length change inverse process to obtain the first processed speech unit sequence; Based on the first probability distribution, determine the speech information corresponding to the mask position that meets the preset filtering conditions in the first processed speech unit sequence, and obtain the second processed speech unit sequence. The mask positions in the second processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

10. The method of claim 9, wherein the functional speech unit includes at least one of a merged speech unit and a blank speech unit; the step of performing a length-changing inverse process on the functional speech unit to obtain a first processed speech unit sequence specifically includes: If the functional voice unit is the merged voice unit, then the functional voice unit is expanded to a preset number of mask positions; If the functional voice unit is a blank voice unit, then the functional voice unit is deleted.

11. The method as described in claim 9, wherein determining the speech information corresponding to the mask position that meets the preset filtering conditions in the first processed speech unit sequence according to the first probability distribution, to obtain the second processed speech unit sequence, specifically includes: Select the mask position to be parsed from the first processed speech unit sequence that meets the preset filtering conditions; Based on the probability distribution of the mask position to be parsed, the speech information corresponding to the mask position to be parsed is determined, and the second processed speech unit sequence is obtained.

12. The method of claim 9, wherein predicting the mask positions in the second processed speech unit sequence to obtain a predicted speech unit sequence specifically includes: The trained speech unit sequence generation model predicts the mask positions in the second processed speech unit sequence to obtain the second probability distribution of each mask position in the second processed speech unit sequence. Based on the second probability distribution, identify the functional speech units at each mask position in the second processed speech unit sequence; The functional speech units at each mask position in the second processed speech unit sequence are subjected to length-changing inverse processing to obtain the third processed speech unit sequence; Based on the second probability distribution, determine the voice information corresponding to the mask position that meets the preset filtering conditions in the third processed voice unit sequence, and obtain the fourth processed voice unit sequence. The mask positions in the fourth processed speech unit sequence are predicted to obtain the predicted speech unit sequence.

13. The method of claim 9, further comprising: Obtain the prompt tone information; The step of inputting the text to be processed into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence based on the text to be processed, specifically includes: The prompt timbre information and the text to be processed are input into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution for each mask position in the preset speech unit sequence based on the prompt timbre information and the text to be processed.

14. A speech synthesis method, comprising: Get the text to be processed; The text to be processed is input into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the method described in any one of claims 1-8; The predicted speech unit sequence is decoded to obtain the audio file of the text to be processed.

15. The method of claim 14, further comprising: Obtain the prompt tone information; The step of inputting the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model specifically includes: The text to be processed and the prompt timbre information are input into the trained speech unit sequence generation model to obtain the predicted speech unit sequence corresponding to the text to be processed, generated by the trained speech unit sequence generation model based on the prompt timbre information.

16. A training device for a speech unit sequence generation model, comprising: The first acquisition module is used to acquire the sample speech unit sequence corresponding to the sample text; The first operation module is used to perform a masking operation on the sample speech unit sequence to obtain the first speech unit sequence. The second operation module is used to perform a length change operation on the first speech unit sequence to obtain a second speech unit sequence; the second speech unit sequence includes functional speech units generated by the length change operation. The third operation module is used to perform a masking operation on the functional speech units in the second speech unit sequence to obtain the third speech unit sequence. The training module is used to train the speech unit sequence generation model to be trained based on the sample text, the sample speech unit sequence, and the third speech unit sequence, so as to obtain the trained speech unit sequence generation model.

17. A speech unit sequence generation apparatus, comprising: The second acquisition module is used to acquire the text to be processed. The first input module is used to input the text to be processed into the trained speech unit sequence generation model, so that the trained speech unit sequence generation model generates a first probability distribution of each mask position in the preset speech unit sequence; the first probability distribution of each mask position is used to represent the probability of predicting the corresponding mask position as any actual speech unit or any functional speech unit; The recognition module is used to identify the functional speech units in each mask position of the preset speech unit sequence according to the first probability distribution by the trained speech unit sequence generation model. The inverse processing module is used to perform length-change inverse processing on the functional speech units to obtain a first processed speech unit sequence; The voice information determination module is used to determine the voice information corresponding to the mask position that meets the preset filtering conditions in the first processed voice unit sequence according to the first probability distribution, so as to obtain the second processed voice unit sequence. The prediction module is used to predict the mask positions in the second processed speech unit sequence to obtain the predicted speech unit sequence.

18. A speech synthesis device, comprising: The third acquisition module is used to acquire the text to be processed; The second input module is used to input the text to be processed into the trained speech unit sequence generation model to obtain the predicted speech unit sequence output by the trained speech unit sequence generation model; the trained speech unit sequence generation model is trained using the method described in any one of claims 1-8. The decoding module is used to decode the predicted speech unit sequence to obtain the audio file of the text to be processed.

19. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 15.

20. A computer-readable storage medium storing computer instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

21. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 15.

Images