Speech synthesis methods, devices, electronic devices and storage media

Abstract

This invention provides a speech synthesis method, apparatus, electronic device, and storage medium. The method includes: extracting text features of the text to be synthesized; searching for a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; and synthesizing speech from the text based on the text features and the target speech code to obtain synthesized speech. The speaker codebook stores speech codes of different speakers, where each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers. The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention significantly reduce model redundancy and improve the utilization rate of model parameters while ensuring the accuracy of speech synthesis.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to a speech synthesis method, apparatus, electronic device, and storage medium. Background Technology

[0002] Speech synthesis aims to convert text information into natural and fluent speech output, and it is widely used in various fields such as virtual assistants, voice navigation, audiobook production, and accessibility devices. A key step in speech synthesis is the effective encoding of the speaker's voice features, which directly affects the naturalness and recognizability of the synthesized speech.

[0003] Current speech synthesis models typically employ vectorization, assigning a unique vector to each speaker to represent their speech features. However, when dealing with massive datasets, such as those containing millions or even tens of millions of different speakers, assigning a unique vector to each speaker leads to an explosive increase in memory consumption. Furthermore, due to the uneven distribution of data, some speakers have ample data while others have severely insufficient data, resulting in low information utilization of these vectors. This increases model redundancy and reduces the overall effectiveness of speech synthesis. Summary of the Invention

[0004] This invention provides a speech synthesis method, apparatus, electronic device, and storage medium to address the deficiencies in the prior art.

[0005] This invention provides a speech synthesis method, comprising the following steps:

[0006] Extract text features from the text to be synthesized;

[0007] Based on the target speaker's identifier, search the speaker codebook for the target speech code that matches the target speaker;

[0008] Based on the text features and the target speech encoding, the text to be synthesized is used to synthesize speech to obtain synthesized speech.

[0009] The speaker codebook stores the speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0010] According to a speech synthesis method provided by the present invention, the speech codes of each speaker are determined based on the following steps:

[0011] The speaker codebook size is determined based on the total number of speakers;

[0012] Based on the speaker codebook size, a subset of codes is selected from multiple basic speech codes of each speaker as the speech code for each speaker.

[0013] According to a speech synthesis method provided by the present invention, the step of selecting a portion of the codes from multiple basic speech codes of each speaker as the speech codes of each speaker based on the size of the speaker's codebook includes:

[0014] Based on the codebook sparsity factor and the speaker's codebook size, the number of partial codes corresponding to each speaker is determined, and a corresponding number of partial codes are randomly selected from multiple basic speech codes of each speaker as the speech codes of each speaker.

[0015] or,

[0016] The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, a portion of the codes are selected from multiple basic speech codes of each speaker as the speech code of each speaker.

[0017] According to a speech synthesis method provided by the present invention, the step of synthesizing speech from the text to be synthesized based on the text features and the target speech code to obtain synthesized speech includes:

[0018] Based on the prior model, the text features and the target speech encoding are applied to generate prior features;

[0019] Based on the speech synthesis model, the text features, the target speech encoding, and the prior features are applied to synthesize the text to be synthesized, thereby obtaining the synthesized speech.

[0020] According to a speech synthesis method provided by the present invention, the speech synthesis model is trained by combining the prior model and the posterior model, and the speech synthesis model is trained based on the following steps:

[0021] Extract sample text features from the sample synthesized text, and search for the first sample speech code that matches the sample speaker in the speaker codebook based on the identifier of the sample speaker;

[0022] Based on the initial model of the prior model, the sample text features and the first sample speech encoding are applied to obtain the first sample prior features;

[0023] Based on the initial model of the posterior model, the synthetic speech features of the synthetic speech tag are restored to obtain the sample posterior features, which have the same dimension as the first sample prior features.

[0024] Based on the initial model of the speech synthesis model, the first sample synthesized speech is obtained by applying the first sample prior features, the sample text features, and the first sample speech encoding.

[0025] Based on the difference between the prior features of the first sample and the posterior features of the sample, and the difference between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model.

[0026] According to a speech synthesis method provided by the present invention, the method further includes, based on the difference between the prior features of the first sample and the posterior features of the sample, and the difference between the synthesized speech of the first sample and the synthesized speech label, jointly updating the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model to obtain the speech synthesis model, the prior model, and the posterior model, and then further comprising:

[0027] Extract sample speech features from the sample speaker, and search for a second sample speech code that matches the sample speech features in the speaker's codebook based on the sample speech features;

[0028] Based on the prior model, the second sample speech encoding and the sample text features are applied to obtain the second sample prior features;

[0029] Based on the speech synthesis model, the sample text features, the second sample prior features, and the second sample speech encoding are input to the model to obtain the second sample synthesized speech.

[0030] Based on the difference between the prior features of the second sample and the posterior features of the sample, as well as the difference between the synthesized speech of the second sample and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0031] According to a speech synthesis method provided by the present invention, the method involves applying the text features, the target speech code, and the prior features based on a speech synthesis model to synthesize the text to be synthesized, thereby obtaining the synthesized speech. The method includes:

[0032] Construct speech synthesis prompt text, which is used to indicate the speech synthesis requirements of the text to be synthesized;

[0033] Based on the speech synthesis model, the speech synthesis prompt text, the text features, the target speech encoding, and the prior features are applied to synthesize the text to be synthesized, thereby obtaining the synthesized speech.

[0034] The present invention also provides a speech synthesis device, comprising the following modules:

[0035] The extraction unit is used to extract the text features of the text to be synthesized;

[0036] The lookup unit is used to search for a target speech code that matches the target speaker in the speaker codebook based on the identifier of the target speaker;

[0037] A synthesis unit is used to perform speech synthesis on the text to be synthesized based on the text features and the target speech code to obtain synthesized speech;

[0038] The speaker codebook stores the speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0039] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement any of the above-described speech synthesis methods.

[0040] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the speech synthesis method as described above.

[0041] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described speech synthesis methods.

[0042] The speech synthesis method, apparatus, electronic device, and storage medium provided by this invention compress the original speaker codebook by selecting partial codes from multiple basic speech codes for each speaker in the speaker codebook. This avoids the problem of directly storing all basic speech codes for each speaker to represent the speech features of each speech, which would result in a large storage space requirement. Furthermore, by selecting partial codes from multiple basic speech codes for each speaker, this invention ensures that each speaker has a relatively balanced set of basic speech codes in the speaker codebook, thus avoiding data redundancy. Further, the total number of partial codes corresponding to all speakers in the speaker codebook is greater than or equal to the total number of speakers, ensuring that each speaker is assigned at least one basic speech code as their speech code, thereby achieving comprehensive data coverage. While maintaining the accuracy of speech synthesis, this invention significantly reduces model redundancy and improves the utilization rate of model parameters. Attached Figure Description

[0043] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0044] Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention.

[0045] Figure 2 This is a schematic diagram of the training process of the speech synthesis model provided by the present invention.

[0046] Figure 3 This is a schematic diagram of the prior and posterior feature extraction model provided by the present invention.

[0047] Figure 4 This is a schematic diagram of the prior feature extraction layer provided by the present invention.

[0048] Figure 5 This is a schematic diagram of the structure of the FFN network provided by the present invention.

[0049] Figure 6 This is a schematic diagram of the posterior feature extraction layer provided by the present invention.

[0050] Figure 7 This is a schematic diagram of the fine-tuning process of the speech synthesis model provided by the present invention.

[0051] Figure 8 This is a schematic diagram of the speech synthesis device provided by the present invention.

[0052] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0053] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0054] Current speech synthesis models typically employ a vectorization approach, assigning each speaker a unique vector (based on multiple base speech codes, each representing different acoustic parameters for the corresponding speaker) to characterize their speech features. However, when dealing with massive datasets containing millions or even tens of millions of different speakers, assigning a unique vector to each speaker leads to an explosive increase in memory usage. Furthermore, due to the uneven distribution of data, some speakers have ample data while others have severely insufficient data. This results in speakers with abundant data having a large number of base speech codes, while those with insufficient data have only a small number. Consequently, during speech synthesis, a significant amount of resources may be allocated to speakers with abundant data, while neglecting those with insufficient data. This not only increases model redundancy and reduces the overall effectiveness of speech synthesis but also leads to uneven utilization of base speech codes.

[0055] For example, for a 2×10 7 Voice data from 10 different speakers, traditional methods use 2×10 7 Representing each speaker's speech features with a separate vector requires a significant amount of memory. Furthermore, the low coupling between different speakers, coupled with the limited amount of speech data for some speakers, leads to considerable model redundancy.

[0056] In response, this invention provides a speech synthesis method. Figure 1 This is a flowchart illustrating the speech synthesis method provided by the present invention, as shown below. Figure 1 As shown, the method includes steps 110, 120, 130, and 140.

[0057] Step 110: Extract the text features of the text to be synthesized.

[0058] Here, the text to be synthesized refers to the text content that needs to be synthesized into speech. The text to be synthesized can be any form of text, such as sentences, paragraphs, or complete articles, depending on the application scenario of speech synthesis. For example, in scenarios such as voice assistants, audiobook generation, and automatic voice notifications, the text to be synthesized can be user-inputted commands, text content from books, or notification information generated by the system.

[0059] Furthermore, text features refer to the semantic, stylistic, and sentiment information extracted from the text to be synthesized. These text features can include linguistic features (such as phonemes, syllables, and parts of speech) and semantic features.

[0060] For example, when extracting text features from the text to be synthesized, text features can be extracted from the text to be synthesized based on a machine learning model, such as the BERT model.

[0061] Step 120: Based on the target speaker's identifier, search for the target speech code that matches the target speaker in the speaker codebook. The speaker codebook stores speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0062] Specifically, the target speaker refers to the person whose speech needs to be synthesized; it can also be understood as the synthesized speech that needs to imitate the voice of the target person when synthesizing text. The identifier of the target speaker is used to represent the identity of the target speaker, and its identifier can be represented by the target speaker's ID.

[0063] Typically, a mapping relationship is established between the identifiers of different speakers and their corresponding speech codes in the speaker codebook. Therefore, after determining the identifier of the target speaker, the target speech code matching the target speaker can be found in the speaker codebook based on this mapping relationship. This target speech code is used to represent the speech feature information of the target speaker, such as pitch, speech rate, intonation, and pronunciation habits. The speaker codebook stores the speech codes of different speakers, and these speech codes are used to represent the speech feature information of the corresponding speakers.

[0064] Furthermore, each speaker's speech features contain a large number of acoustic parameters and details, which need to be fully described through a series of basic speech codes. However, when constructing a speaker codebook, directly storing all the basic speech codes of each speaker to represent the speech features of each speech would not only occupy a huge amount of storage space, but also increase the complexity of data processing and the latency of real-time synthesis.

[0065] Considering that some basic speech codes may contain repetitive or similar information when describing a speaker's speech characteristics, this information is not necessary for distinguishing different speakers or synthesizing high-quality speech. To address this, a portion of the basic speech codes corresponding to each speaker can be randomly selected as that speaker's speech code. This can reduce the aforementioned redundant information to some extent, thereby lowering storage and processing costs.

[0066] Furthermore, when constructing the speaker codebook, it's important to consider that some speakers may possess a large number of basic speech codes, while others may have only a small number. This imbalance not only affects the training performance of the speech synthesis model but can also lead to inconsistent quality of synthesized speech, especially for speakers with limited data. This invention addresses this by selecting a subset of codes from each speaker's multiple basic speech codes as their corresponding speech codes. This ensures that each speaker has a relatively balanced set of basic speech codes in the speaker codebook, thereby avoiding data redundancy.

[0067] Furthermore, considering that each speaker's speech features are unique, containing specific acoustic parameters, intonation patterns, pronunciation habits, and other key information, to ensure that these unique speech features can be fully captured and accurately reproduced by the system, at least one basic speech code needs to be assigned to each speaker as their speech code. When the total number of corresponding partial codes for all speakers is greater than or equal to the total number of speakers, it can be ensured that each speaker is assigned at least one basic speech code as their speech code, thereby achieving comprehensive data coverage.

[0068] Step 130: Based on text features and target speech encoding, perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0069] Specifically, text features are used to characterize the linguistic information contained in the text to be synthesized, and target encoding is used to characterize the speech features of the target speaker. Based on the text features, the semantic content and expression style of the synthesized speech can be determined. At the same time, based on the target speech encoding, the acoustic parameters of the synthesized speech can be adjusted to generate synthesized speech that conforms to the style of the target speaker.

[0070] Optionally, after obtaining the synthesized speech, post-processing can be performed on the generated synthesized speech, including steps such as noise removal and sound quality optimization, to improve the naturalness and fluency of the synthesized speech.

[0071] In the speech synthesis method provided by the embodiments of the present invention, the speech encodings of each speaker in the speaker codebook are partial encodings selected from multiple basic speech encodings of each speaker, avoiding the problem of occupying a large storage space caused by directly storing all the basic speech encodings of each speaker to represent the speech features of each speaker, and realizing the compression of the original speaker codebook. In addition, in the embodiments of the present invention, by selecting partial encodings from multiple basic speech encodings of each speaker as the speech encodings of the corresponding speaker, it can be ensured that each speaker has relatively balanced basic speech encodings in the speaker codebook, thereby avoiding data redundancy. Further, the total number of partial encodings corresponding to all speakers in the speaker codebook is greater than or equal to the number of speakers, so as to ensure that each speaker is assigned at least one basic speech encoding as the speech encoding, thus realizing the comprehensiveness of data coverage. On the basis of ensuring the speech synthesis accuracy, the model redundancy is significantly reduced, and the utilization rate of the model parameters is improved.

[0072] Based on the above embodiments, the speech encodings of each speaker are determined based on the following steps:

[0073] Based on the total number of speakers, determine the size of the speaker codebook;

[0074] Based on the size of the speaker codebook, select partial encodings from multiple basic speech encodings of each speaker as the speech encodings of each speaker.

[0075] Specifically, the size of the speaker codebook is used to represent the number of basic speech encodings that can be stored. To ensure that the speaker codebook can accommodate the basic speech encodings of all speakers, in the embodiments of the present invention, the size of the speaker codebook is determined based on the total number of speakers. After determining the size of the speaker codebook, the upper limit of the basic speech encodings that the speaker codebook can accommodate can be determined, that is, the sum of the basic speech encodings of all speakers needs to be less than or equal to the upper limit of the basic speech encodings that the speaker codebook can accommodate.

[0076] On the basis of ensuring that the sum of the basic speech encodings of all speakers needs to be less than or equal to the upper limit of the basic speech encodings that the speaker codebook can accommodate, select partial encodings from multiple basic speech encodings of each speaker as the speech encodings of each speaker.

[0077] For example, each speaker includes n basic speech encodings, and m basic speech encodings can be selected from them as the speech encoding of each speaker (m < n). When selecting m basic speech encodings, clustering analysis can be performed based on the similarity between the basic speech encodings in the n basic speech encodings of the corresponding speaker, dividing the n basic speech encodings into m groups, and randomly selecting one basic speech encoding from each group as the speech encoding of the speaker.

[0078] Based on any of the above embodiments, selecting some encodings from multiple basic voice encodings of each speaker as the voice encoding of each speaker according to the size of the speaker codebook includes:

[0079] Determining the number of corresponding partial encodings of each speaker based on the codebook sparsity factor and the size of the speaker codebook, and randomly selecting the corresponding number of partial encodings from multiple basic voice encodings of each speaker as the voice encoding of each speaker;

[0080] Or,

[0081] Randomly sorting each speaker, and selecting some encodings from multiple basic voice encodings of each speaker as the voice encoding of each speaker based on the binary representation corresponding to the sorting serial number of each speaker.

[0082] Specifically, the codebook sparsity rate is used to represent the proportion of the number of corresponding partial encodings of each speaker in the number of corresponding basic voice encodings. For example, each speaker includes n basic voice encodings, and m basic voice encodings are selected from the n basic voice encodings of each speaker as its voice encoding, then the codebook sparsity factor = m / n.

[0083] In addition, the size of the speaker codebook is used to represent the number of basic voice encodings that the codebook can store for each speaker, that is, the size of the speaker codebook can be determined by the number of basic voice encodings of each speaker in the speaker codebook.

[0084] Based on this, according to the codebook sparsity factor and the size of the speaker codebook, the number of corresponding partial encodings of each speaker can be determined, and the corresponding number of partial encodings can be randomly selected from multiple basic voice encodings of each speaker as the voice encoding of each speaker.

[0085] Suppose there are 3 million speakers, and n basic voice encodings with a dimension of 1024 are required for representation. The voice encoding of each speaker is represented by m basic voice encodings among the n basic voice encodings. Define the codebook sparsity factor r = m / n. Usually, the codebook sparsity factor is set to about 5%. Therefore, for a speaker codebook of size n×1024, the number of speakers that can be represented is as follows:

[0086]

[0087] For 3 million speakers, it is necessary to satisfy >3000000. Here, select n = 95, m = 4, calculate P(95, 4) = <3183545> 3000000, and the codebook sparsity factor r = m / n = 4 / 95 = 4.2% to ensure that each speaker has the same number of basic voice encodings for representation.

[0088] For example, when encoding the speech data of 10 speakers, based on the method of the above embodiment, the size of the speaker codebook is n = 5, that is, each person in the speaker codebook includes 5 basic speech encodings, and the number m of valid basic speech encodings selected from the 5 basic speech encodings is 2. Table 1 is one of the speech encoding information lists. As shown in Table 1, each speaker includes 5 basic speech encodings. "1" indicates the selected valid basic speech encoding. For example, for the first speaker, the first and second basic speech encodings are selected as the valid basic speech encodings, and the remaining digits are "0", that is, the basic speech encodings of the remaining digits are not included in the speaker codebook.

[0089] Table 1

[0090]

[0091] In addition, as another alternative, each speaker can be randomly sorted, and based on the binary representation corresponding to the sorting serial number of each speaker, a part of the encodings are selected from the multiple basic speech encodings of each speaker as the speech encoding of each speaker. Among them, it can be determined whether to select the corresponding basic speech encoding according to the value of each bit in the binary representation. For example, if a certain bit in the binary representation is "1", it means to select the basic speech encoding at the corresponding digit, and if a certain bit in the binary representation is "0", it means not to select the basic speech encoding at the corresponding digit.

[0092] Among them, the size of the codebook can be determined based on the number of bits of the binary representation. For example, if the number of bits of the binary representation is b, the size of the speaker codebook is b × 1024, and the upper limit of the basic speech encodings that the corresponding speaker codebook can accommodate is . To achieve data full coverage, it is necessary to satisfy , where represents the number of speakers.

[0093] For example, if there are 3,000,000 speakers, b = 22 can satisfy = 4,194,303 > 3,000,000, so that the size of the speaker codebook can be further compressed, and the number of partial encodings selected by each speaker is different.

[0094] Table 2 is the second list of speech coding information. As shown in Table 2, the speaker codebook size b=4, meaning that the effective basic speech codes of all speakers under the first position, the second position, the third position, and the fourth position are stored in four separate speaker codebooks. The binary representation of the first speaker's number "1" is "1000", meaning the first bit is "1", which means the first basic speech code is selected as the effective basic speech code, and the remaining bits are "0", meaning the basic speech codes of the remaining bits are not included in the speaker codebook.

[0095] Table 2

[0096]

[0097] To ensure the hit probability of the basic speech codes in each speaker's codebook, the basic speech codes can be randomly shuffled, and then random selection can be made from the shuffled basic speech codes. Furthermore, the effective speech codes can be normalized to ensure that each effective speech code contributes equally to speech synthesis.

[0098] Based on any of the above embodiments, based on text features and target speech encoding, speech synthesis is performed on the text to be synthesized to obtain synthesized speech, including:

[0099] Based on the prior model, text features and target speech encoding are applied to generate prior features;

[0100] Based on the speech synthesis model, text features, target speech encoding, and prior features are applied to synthesize the text to be synthesized, resulting in synthesized speech.

[0101] Specifically, prior features are intermediate features generated by a prior model based on text features and target speech encoding before speech synthesis. They can capture the inherent attributes of speech (such as rhythm, intonation, pronunciation habits, etc.).

[0102] Furthermore, prior features contain the correlation information between text features and target speech encoding (such as the correspondence between text content and speech features, the correspondence between emotional information and text content, and the pronunciation relationship between adjacent words), enabling speech synthesis models to capture these features more accurately when generating speech, thereby improving the accuracy of speech synthesis.

[0103] This invention, through the generation of prior features followed by speech synthesis based on these prior features, text features, and target speech encoding, fully leverages prior features to enhance the speech synthesis effect. By capturing the inherent properties of speech through prior features, improving the accuracy of speech synthesis, and enhancing the model's generalization ability, it is possible to generate more natural, fluent, and human-pleasing synthesized speech.

[0104] Based on any of the above embodiments, the speech synthesis model is trained by jointly using the prior model and the posterior model. The speech synthesis model is trained based on the following steps:

[0105] Extract the sample text features of the synthesized sample text, and based on the identifier of the sample speaker, search for the first sample speech code that matches the sample speaker in the speaker codebook;

[0106] Based on the initial model of the prior model, the sample text features and the first sample speech encoding are applied to obtain the first sample prior features;

[0107] Based on the initial model of the posterior model, the synthetic speech features of the synthetic speech tag are restored to obtain the sample posterior features. The sample posterior features have the same dimension as the first sample prior features.

[0108] Based on the initial model of the speech synthesis model, the first sample synthesized speech is obtained by applying the prior features of the first sample, the features of the sample text, and the speech encoding of the first sample.

[0109] Based on the differences between the prior features and posterior features of the first sample, and the differences between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model.

[0110] Specifically, the prior model is used to capture the correlation information between the features of the sample text and the first sample speech code, as well as the inherent speech attributes in the first sample speech code. This provides reliable prior knowledge for speech synthesis, guiding the speech synthesis model to synthesize speech from the sample text. The posterior model is used to perform feature reconstruction on the synthesized speech features, obtaining sample posterior features with the same dimension as the first sample prior features. The synthesized speech features can be obtained from a pre-defined speech codebook based on the identifier of the sample speaker.

[0111] The difference between the first sample prior features and the sample posterior features reflects the prediction accuracy of the prior model for the prior features. The smaller the difference, the smaller the difference between the first sample prior features predicted by the prior model and the actual features (i.e., the sample posterior features), and thus the more accurately the prior model can predict the prior features. By minimizing the difference between the first sample prior features and the sample posterior features as the training objective, the prior model can be trained to more accurately capture the correlation information between text and speech, as well as the inherent properties of speech, thereby enabling it to predict the prior features more accurately.

[0112] Furthermore, the difference between the first sample synthesized speech and the synthesized speech label is used to characterize the difference between the synthesized speech output by the speech synthesis model and the actual speech. With minimizing the difference between the first sample synthesized speech and the synthesized speech label as the training objective, the speech synthesis model can be trained to generate synthesized speech that matches the actual speech more accurately, thereby improving the naturalness and realism of the synthesized speech.

[0113] Figure 2 This is a schematic diagram of the training process of the speech synthesis model provided by the present invention, as shown below. Figure 2 As shown, in the preset speech codec, the synthesized speech features are found based on the speaker identifier (speech_codec) of the sample, and the synthesized speech features are input into the prior and posterior feature extraction model (deep_attention) to obtain the sample posterior features.

[0114] Simultaneously, based on the speaker identifier (spk_id), the first sample speech code matching the speaker is searched in the speaker codebook (spk_codebook), and the first sample speech code is input into the prior and posterior feature extraction model (deep_attention). The sample text features (text_encode_input) of the synthesized sample text are also input into the prior and posterior feature extraction model (deep_attention) to obtain the first sample prior features.

[0115] The first sample prior features, sample text features, and first sample speech encoding are input into the speech synthesis model to obtain the first sample synthesized speech;

[0116] Based on the differences between the prior features and posterior features of the first sample, and the differences between the synthesized speech of the first sample and the synthesized speech label, the parameters of the prior and posterior feature extraction model (deep_attention) and the speech synthesis model are jointly updated to obtain the trained speech synthesis model.

[0117] Figure 3 This is a schematic diagram of the prior and posterior feature extraction model provided by the present invention, as shown below. Figure 2 As shown, the prior and posterior feature extraction model (deep_attention) includes a prior model and a posterior model. The prior model consists of a prior feature extraction layer (map_attention), a normalization layer (layernorm), an FFN network, and another normalization layer (layernorm). After inputting the sample text features and the first sample speech encoding (y_feature) into the prior feature extraction layer, the data passes through the normalization layer, the FFN network, and another normalization layer to obtain the first sample prior feature (y_output).

[0118] The posterior model consists of a prior feature extraction layer (map_attention), a normalization layer (layernorm), an FFN network, a normalization layer (layernorm), a first posterior feature extraction layer (map_mean_layer), and a second posterior feature extraction layer (map_var_layer). The synthesized speech features (x_feature) are input to the prior feature extraction layer (map_attention), then pass through the normalization layer (layernorm), the FFN network, the normalization layer (layernorm), the first posterior feature extraction layer (map_mean_layer), and the second posterior feature extraction layer (map_var_layer) to obtain the sample posterior features (x_output). The loss value is determined based on the first sample prior feature (y_output) and the sample posterior feature (x_output).

[0119] Figure 4 This is a schematic diagram of the prior feature extraction layer provided by the present invention, as shown below. Figure 4As shown, the prior feature extraction layer (map_attention) includes a key convolutional network (conv1d_k), a value convolutional network (conv1d_v), a matmul module, a drop layer, and a softmax layer. The prior feature extraction process of this layer is as follows: The input feature (input_feature) is fed into the key convolutional network (conv1d_k) and the value convolutional network (conv1d_v) respectively to obtain the corresponding convolutional features. Then, the two convolutional features are multiplied to obtain a fixed-size matrix feature, which is the codebook feature. The codebook feature is a fixed-dimensional feature and is independent of the time dimension feature. For example, assuming that the dimension of the input feature is B×T×C (where B represents the size of the batch of data, T is the time dimension feature, and C is the extracted fixed-dimensional feature), the key feature and value feature can be obtained by the key convolutional network (conv1d_k) and the value convolutional network (conv1d_v) respectively, with the corresponding feature dimensions of B×T×C. Then, the matrix is ​​input into the matmul module for matrix multiplication to obtain the fixed codebook feature with a feature dimension of B×C×C. Finally, the fixed codebook feature is processed by the drop layer and the softmax layer to obtain the final codebook feature output_feature.

[0120] Figure 5 This is a schematic diagram of the FFN network structure provided by the present invention, as shown below. Figure 5 As shown, the execution flow of the FFN network is as follows: First, the input feature (input_feature) is input into the convolutional upsampling network (conv1d_upsample) to upsample the C dimension to a 4×C dimension. Then, the upsampled feature is processed by the activation function and input into the convolutional downsampling network (conv1d_downsample) to restore the 4×C dimension feature to a C dimension feature. Finally, the restored feature is output.

[0121] Figure 6 This is a schematic diagram of the posterior feature extraction layer provided by the present invention, as shown below. Figure 6 As shown, the posterior feature extraction layer (map_mean_layer and map_var_layer have the same structure) first inputs the input feature into a linear upsampling network (linear_upsample) to upsample the C-dimensional feature to 4×C. Then, the upsampled feature is input into a ReLU activation function for feature transformation. Finally, the transformed feature is input into a linear downsampling network (linear_downsample) for downsampling to obtain a C-dimensional feature, which is then output.

[0122] Based on any of the above embodiments, based on the difference between the prior features and posterior features of the first sample, and the difference between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model, and then the process further includes:

[0123] Extract the sample speech features of the sample speaker, and search for the second sample speech code that matches the sample speech features in the speaker codebook based on the sample speech features;

[0124] Based on the prior model, the second sample speech coding and sample text features are applied to obtain the second sample prior features;

[0125] Based on the speech synthesis model, the sample text features, the second sample prior features, and the second sample speech encoding are input to obtain the second sample synthesized speech.

[0126] Based on the differences between the prior and posterior features of the second sample, and the differences between the synthesized speech of the second sample and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0127] Specifically, based on the differences between the prior features and posterior features of the first sample, and the differences between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial models of the prior model, the initial models of the posterior model, and the initial models of the speech synthesis model are jointly updated. After obtaining the speech synthesis model, the prior model, and the posterior model, the parameters of the speech synthesis model and the prior model can be fine-tuned so that the speech synthesis model and the prior model can generate synthesized speech of a specific sample speaker. It should be noted that, in order to preserve as much of the underlying model compression information from the above-mentioned large-scale training as possible, this embodiment of the invention, after obtaining the speech synthesis model, the prior model, and the posterior model, fixes the parameters of the posterior model and fine-tunes the parameters of the speech synthesis model and the prior model.

[0128] Figure 7 This is a schematic diagram of the fine-tuning process of the speech synthesis model provided by the present invention, as shown below. Figure 7 As shown, the fine-tuning process is as follows: extract the sample speech features (spk_embed) of the sample speaker, and search for the second sample speech code that matches the sample speech features in the speaker codebook (spk_codebook) based on the sample speech features. Input the second sample speech code (text_encode_input) and the sample text features into the prior and posterior feature extraction model (deep_attention) to obtain the second sample prior features.

[0129] In the preset speech codec, the synthesized speech features are found based on the speaker identifier (speech_codec) of the sample and then input into the prior and posterior feature extraction model (deep_attention) to obtain the sample posterior features.

[0130] The sample text features, the second sample prior features, and the second sample speech code are input into the speech synthesis model to obtain the second sample synthesized speech (speech_codec). Based on the differences between the second sample prior features and the sample posterior features, as well as the differences between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0131] Based on any of the above embodiments, based on a speech synthesis model, text features, target speech encoding, and prior features are applied to perform speech synthesis on the text to be synthesized, resulting in synthesized speech, including:

[0132] Construct speech synthesis prompt text, which is used to indicate the speech synthesis requirements of the text to be synthesized;

[0133] Based on the speech synthesis model, the speech synthesis prompt text, text features, target speech encoding, and prior features are applied to synthesize the text to be synthesized, resulting in synthesized speech.

[0134] Specifically, different texts may require different speech styles, speeds, and intonations. For example, news broadcasts might need a formal, steady tone, while children's stories might require a friendly, lively tone. Speech synthesis prompts can explicitly indicate these requirements, enabling the speech synthesis model to generate appropriate speech based on different needs.

[0135] By incorporating speech synthesis requirements for indicating the text to be synthesized, this invention not only improves the accuracy and naturalness of speech synthesis but also meets diverse speech synthesis needs, thereby enhancing the user experience.

[0136] like Figure 2 and Figure 7 As shown, during the training and fine-tuning of the model, the speech synthesis prompt text (prompt_speech_codec) can also be input into the speech synthesis model to guide the speech synthesis model to generate synthesized speech that meets the requirements.

[0137] The speech synthesis apparatus provided by the present invention will be described below. The speech synthesis apparatus described below can be referred to in correspondence with the speech synthesis method described above.

[0138] Based on any of the above embodiments Figure 8This is a schematic diagram of the speech synthesis device provided by the present invention, as shown below. Figure 8 As shown, the device includes:

[0139] Extraction unit 810 is used to extract text features of the text to be synthesized;

[0140] The lookup unit 820 is used to search for a target speech code that matches the target speaker in the speaker codebook based on the identifier of the target speaker;

[0141] Synthesis unit 830 is used to synthesize speech from text to be synthesized based on text features and target speech encoding to obtain synthesized speech.

[0142] The speaker codebook stores the speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0143] Based on any of the above embodiments, the speech coding of each speaker is determined based on the following steps:

[0144] The speaker codebook size is determined based on the total number of speakers;

[0145] Based on the speaker's codebook size, a subset of codes is selected from multiple basic speech codes for each speaker as their speech codes.

[0146] Based on any of the above embodiments, and based on the speaker codebook size, a subset of codes is selected from multiple basic speech codes of each speaker as the speech code for each speaker, including:

[0147] Based on the codebook sparsity factor and the speaker's codebook size, the number of corresponding partial codes for each speaker is determined, and a corresponding number of partial codes are randomly selected from multiple basic speech codes of each speaker as the speech codes of each speaker.

[0148] or,

[0149] The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, a portion of the codes are selected from multiple basic speech codes of each speaker as the speech code of each speaker.

[0150] Based on any of the above embodiments, based on text features and target speech encoding, speech synthesis is performed on the text to be synthesized to obtain synthesized speech, including:

[0151] Based on the prior model, text features and target speech encoding are applied to generate prior features;

[0152] Based on the speech synthesis model, text features, target speech encoding, and prior features are applied to synthesize the text to be synthesized, resulting in synthesized speech.

[0153] Based on any of the above embodiments, the speech synthesis model is trained by jointly using the prior model and the posterior model. The speech synthesis model is trained based on the following steps:

[0154] Extract the sample text features of the synthesized sample text, and based on the identifier of the sample speaker, search for the first sample speech code that matches the sample speaker in the speaker codebook;

[0155] Based on the initial model of the prior model, the sample text features and the first sample speech encoding are applied to obtain the first sample prior features;

[0156] Based on the initial model of the posterior model, the synthetic speech features of the synthetic speech tag are restored to obtain the sample posterior features. The sample posterior features have the same dimension as the first sample prior features.

[0157] Based on the initial model of the speech synthesis model, the first sample synthesized speech is obtained by applying the prior features of the first sample, the features of the sample text, and the speech encoding of the first sample.

[0158] Based on the differences between the prior features and posterior features of the first sample, and the differences between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model.

[0159] Based on any of the above embodiments, based on the difference between the prior features and posterior features of the first sample, and the difference between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model, and then the process further includes:

[0160] Extract the sample speech features of the sample speaker, and search for the second sample speech code that matches the sample speech features in the speaker codebook based on the sample speech features;

[0161] Based on the prior model, the second sample speech coding and sample text features are applied to obtain the second sample prior features;

[0162] Based on the speech synthesis model, the sample text features, the second sample prior features, and the second sample speech encoding are input to obtain the second sample synthesized speech.

[0163] Based on the differences between the prior and posterior features of the second sample, and the differences between the synthesized speech of the second sample and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0164] Based on any of the above embodiments, based on a speech synthesis model, text features, target speech encoding, and prior features are applied to perform speech synthesis on the text to be synthesized, resulting in synthesized speech, including:

[0165] Construct speech synthesis prompt text, which is used to indicate the speech synthesis requirements of the text to be synthesized;

[0166] Based on the speech synthesis model, the speech synthesis prompt text, text features, target speech encoding, and prior features are applied to synthesize the text to be synthesized, resulting in synthesized speech.

[0167] Figure 9 This is a schematic diagram of the structure of the electronic device provided by the present invention, such as... Figure 9 As shown, the electronic device may include: a processor 910, a communications interface 920, a memory 930, and a communication bus 940, wherein the processor 910, the communications interface 920, and the memory 930 communicate with each other through the communication bus 940. The processor 910 can call logical instructions in the memory 930 to execute a speech synthesis method, which includes: extracting text features of the text to be synthesized; searching for a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; and synthesizing the text to be synthesized based on the text features and the target speech code to obtain synthesized speech; the speaker codebook stores speech codes of different speakers, and each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0168] Furthermore, the logical instructions in the aforementioned memory 930 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0169] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer is able to execute the speech synthesis method provided by the above methods. The method includes: extracting text features of the text to be synthesized; searching for a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; and performing speech synthesis on the text to be synthesized based on the text features and the target speech code to obtain synthesized speech. The speaker codebook stores speech codes of different speakers, and the speech code of each speaker is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0170] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program implements the speech synthesis method provided by the above methods. The method includes: extracting text features of a text to be synthesized; searching for a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; and performing speech synthesis on the text to be synthesized based on the text features and the target speech code to obtain synthesized speech. The speaker codebook stores speech codes of different speakers, wherein each speaker's speech code is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0171] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0172] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0173] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that, include: Extract text features from the text to be synthesized; Based on the target speaker's identifier, search the speaker codebook for the target speech code that matches the target speaker; Based on the text features and the target speech encoding, the text to be synthesized is used to synthesize speech to obtain synthesized speech. The speaker codebook stores the speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of the corresponding speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers. Each basic speech code is used to represent different acoustic parameter information of the corresponding speaker. When the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers, each speaker is assigned at least one basic speech code as its speech code.

2. The speech synthesis method according to claim 1, characterized in that, The speech codes for each speaker are determined based on the following steps: The speaker codebook size is determined based on the total number of speakers; Based on the speaker codebook size, a subset of codes is selected from multiple basic speech codes of each speaker as the speech code for each speaker.

3. The speech synthesis method according to claim 2, characterized in that, The step of selecting a portion of the code from multiple basic speech codes of each speaker based on the speaker's codebook size includes: Based on the codebook sparsity factor and the speaker's codebook size, the number of partial codes corresponding to each speaker is determined, and a corresponding number of partial codes are randomly selected from multiple basic speech codes of each speaker as the speech codes of each speaker. or, The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, a portion of the codes are selected from multiple basic speech codes of each speaker as the speech code of each speaker.

4. The speech synthesis method according to any one of claims 1 to 3, characterized in that, The step of synthesizing speech from the text to be synthesized based on the text features and the target speech encoding to obtain synthesized speech includes: Based on the prior model, the text features and the target speech encoding are applied to generate prior features; Based on the speech synthesis model, the text features, the target speech encoding, and the prior features are applied to synthesize the text to be synthesized, thereby obtaining the synthesized speech.

5. The speech synthesis method according to claim 4, characterized in that, The speech synthesis model is trained by combining the prior model and the posterior model, and the speech synthesis model is trained based on the following steps: Extract sample text features from the sample synthesized text, and search for the first sample speech code that matches the sample speaker in the speaker codebook based on the identifier of the sample speaker; Based on the initial model of the prior model, the sample text features and the first sample speech encoding are applied to obtain the first sample prior features; Based on the initial model of the posterior model, the synthetic speech features of the synthetic speech tag are restored to obtain the sample posterior features, which have the same dimension as the first sample prior features. Based on the initial model of the speech synthesis model, the first sample synthesized speech is obtained by applying the first sample prior features, the sample text features, and the first sample speech encoding. Based on the difference between the prior features of the first sample and the posterior features of the sample, and the difference between the synthesized speech of the first sample and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model.

6. The speech synthesis method according to claim 5, characterized in that, The steps involve jointly updating the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model based on the difference between the prior features and the posterior features of the first sample, and the difference between the synthesized speech and the synthesized speech label of the first sample, to obtain the speech synthesis model, the prior model, and the posterior model. The process further includes: Extract sample speech features from the sample speaker, and search for a second sample speech code that matches the sample speech features in the speaker's codebook based on the sample speech features; Based on the prior model, the second sample speech encoding and the sample text features are applied to obtain the second sample prior features; Based on the speech synthesis model, the sample text features, the second sample prior features, and the second sample speech encoding are input to the model to obtain the second sample synthesized speech. Based on the difference between the prior features of the second sample and the posterior features of the sample, as well as the difference between the synthesized speech of the second sample and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

7. The speech synthesis method according to claim 4, characterized in that, The method, based on a speech synthesis model, applies the text features, the target speech encoding, and the prior features to perform speech synthesis on the text to be synthesized, obtaining the synthesized speech, including: Construct speech synthesis prompt text, which is used to indicate the speech synthesis requirements of the text to be synthesized; Based on the speech synthesis model, the speech synthesis prompt text, the text features, the target speech encoding, and the prior features are applied to synthesize the text to be synthesized, thereby obtaining the synthesized speech.

8. A speech synthesis device, characterized in that, include: The extraction unit is used to extract the text features of the text to be synthesized; The lookup unit is used to search for a target speech code that matches the target speaker in the speaker codebook based on the identifier of the target speaker; A synthesis unit is used to perform speech synthesis on the text to be synthesized based on the text features and the target speech code to obtain synthesized speech; The speaker codebook stores the speech codes of different speakers. Each speaker's speech code is a partial code selected from multiple basic speech codes of the corresponding speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers. Each basic speech code is used to represent different acoustic parameter information of the corresponding speaker. When the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers, each speaker is assigned at least one basic speech code as its speech code.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the speech synthesis method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the speech synthesis method as described in any one of claims 1 to 7.

Images