Speech synthesis method and device, electronic equipment and storage medium

Abstract

The invention provides a speech synthesis method and device, electronic equipment and a storage medium. The method comprises the steps of extracting text features of a to-be-synthesized text; based on the identifier of the target speaker, searching a target voice code matched with the target speaker in a speaker codebook; based on the text feature and the target voice code, performing voice synthesis on the to-be-synthesized text to obtain a synthesized voice; voice codes of different speakers are stored in the speaker codebook, the voice code of each speaker is a partial code selected from a plurality of basic voice codes of each speaker, and the total number of the partial codes corresponding to all the speakers is greater than or equal to the total number of the speakers. According to the speech synthesis method and device, the electronic equipment and the storage medium provided by the invention, on the basis of ensuring the speech synthesis precision, the model redundancy is remarkably reduced, and the parameter utilization rate of the model is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of natural language processing, and in particular to a speech synthesis method, device, electronic device and storage medium. Background Art

[0002] Speech synthesis aims to convert text information into natural and fluent speech output. It is widely used in many fields such as virtual assistants, voice navigation, audio book production, and barrier-free assistive devices. A key link in speech synthesis is the effective encoding of the speaker's voice features, which is directly related to the naturalness and recognizability of the synthesized speech.

[0003] Current speech synthesis models usually use vectorization methods to assign an independent vector to each speaker to represent their speech features. However, when faced with massive data sets, such as speech data containing millions or even tens of millions of different speakers, assigning an independent vector to each speaker directly leads to an explosive growth in memory usage. In addition, due to the uneven distribution of data, some speakers have sufficient data, while others have insufficient data, resulting in low information utilization of these vectors, which in turn increases the redundancy of the model and reduces the overall effect of speech synthesis. Summary of the invention

[0004] The present invention provides a speech synthesis method, device, electronic equipment and storage medium to solve the defects in the prior art.

[0005] The present invention provides a speech synthesis method, comprising the following steps: Extracting text features of the text to be synthesized; Based on the identifier of the target speaker, searching the speaker codebook for a target speech code that matches the target speaker; Based on the text features and the target speech coding, performing speech synthesis on the text to be synthesized to obtain synthesized speech; The speaker codebook stores speech codes of different speakers, wherein the speech code of each speaker is a partial code selected from a plurality of basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0006] According to a speech synthesis method provided by the present invention, the speech coding of each speaker is determined based on the following steps: Determining a speaker codebook size based on the total number of speakers; Based on the speaker codebook size, some codes are selected from a plurality of basic speech codes of each speaker as the speech code of each speaker.

[0007] According to a speech synthesis method provided by the present invention, based on the size of the speaker codebook, selecting part of the codes from the multiple basic speech codes of each speaker as the speech code of each speaker includes: Determining the number of partial codes corresponding to each speaker based on the codebook sparsity factor and the speaker codebook size, and randomly selecting a corresponding number of partial codes from a plurality of basic speech codes of each speaker as the speech code of each speaker; or, The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

[0008] According to a speech synthesis method provided by the present invention, based on the text features and the target speech coding, speech synthesis is performed on the text to be synthesized to obtain synthesized speech, including: Based on the prior model, applying the text features and the target speech coding to generate prior features; Based on the speech synthesis model, the text features, the target speech coding and the prior features are applied to perform speech synthesis on the text to be synthesized to obtain the synthesized speech.

[0009] According to a speech synthesis method provided by the present invention, the speech synthesis model is trained by combining the prior model and the posterior model, and the speech synthesis model is trained based on the following steps: Extracting sample text features of the sample synthesized text, and searching the speaker codebook for a first sample speech code matching the sample speaker based on an identifier of the sample speaker; Based on the initial model of the prior model, applying the sample text feature and the first sample speech coding to obtain a first sample prior feature; Based on the initial model of the posterior model, feature restoration is performed on the synthesized speech features of the synthesized speech label to obtain sample posterior features, where the sample posterior features have the same dimension as the first sample a priori features; Based on the initial model of the speech synthesis model, applying the first sample prior features, the sample text features and the first sample speech coding to obtain a first sample synthesized speech; Based on the difference between the first sample prior features and the sample posterior features, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model and the posterior model.

[0010] According to a speech synthesis method provided by the present invention, based on the difference between the first sample prior feature and the sample posterior feature, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model and the posterior model, and then the method further includes: Extracting a sample speech feature of a sample speaker, and searching the speaker codebook for a second sample speech code that matches the sample speech feature based on the sample speech feature; Based on the prior model, applying the second sample speech coding and the sample text feature to obtain a second sample prior feature; Based on the speech synthesis model, applying the sample text feature, the second sample prior feature and the second sample speech coding input to the to obtain a second sample synthesized speech; Based on the difference between the second sample prior feature and the sample posterior feature, and the difference between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0011] According to a speech synthesis method provided by the present invention, the speech synthesis model is based on the text feature, the target speech coding and the prior feature, the text to be synthesized is subjected to speech synthesis to obtain the synthesized speech, including: Constructing a speech synthesis prompt text, wherein the speech synthesis prompt text is used to indicate the speech synthesis requirement of the text to be synthesized; Based on the speech synthesis model, the speech synthesis prompt text, the text features, the target speech coding and the prior features are applied to perform speech synthesis on the text to be synthesized to obtain the synthesized speech.

[0012] The present invention also provides a speech synthesis device, comprising the following modules: An extraction unit, used for extracting text features of the text to be synthesized; A search unit, configured to search a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; A synthesis unit, configured to perform speech synthesis on the text to be synthesized based on the text features and the target speech coding to obtain synthesized speech; The speaker codebook stores speech codes of different speakers, wherein the speech code of each speaker is a partial code selected from a plurality of basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned speech synthesis methods is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis method described in any one of the above is implemented.

[0015] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements any of the above-mentioned speech synthesis methods.

[0016] In the speech synthesis method, device, electronic device and storage medium provided by the present invention, the speech code of each speaker in the speaker codebook is a partial code selected from multiple basic speech codes of each speaker, avoiding the problem of directly storing all the basic speech codes of each speaker to represent the speech characteristics of each speech, resulting in a large storage space, and realizing the compression of the original speaker codebook. In addition, the present invention selects a partial code from multiple basic speech codes of each speaker as the speech code of the corresponding speaker, thereby ensuring that each speaker has a relatively balanced basic speech code in the speaker codebook, thereby avoiding data redundancy. Furthermore, the total number of partial codes corresponding to all speakers in the speaker codebook is greater than or equal to the total number of speakers, thereby ensuring that each speaker is assigned at least one basic speech code as the speech code, thereby realizing the comprehensiveness of data coverage. On the basis of ensuring the accuracy of speech synthesis, the model redundancy is significantly reduced, and the utilization rate of the model parameters is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 It is a flow chart of the speech synthesis method provided by the present invention.

[0019] Figure 2 It is a schematic diagram of the training process of the speech synthesis model provided by the present invention.

[0020] Figure 3 It is a structural schematic diagram of the a priori and a posteriori feature extraction model provided by the present invention.

[0021] Figure 4It is a structural schematic diagram of the prior feature extraction layer provided by the present invention.

[0022] Figure 5 It is a schematic diagram of the structure of the FFN network provided by the present invention.

[0023] Figure 6 It is a structural schematic diagram of the posterior feature extraction layer provided by the present invention.

[0024] Figure 7 It is a schematic diagram of the fine-tuning process of the speech synthesis model provided by the present invention.

[0025] Figure 8 It is a structural schematic diagram of the speech synthesis device provided by the present invention.

[0026] Fig. 9 It is a structural schematic diagram of the electronic device provided by the present invention. DETAILED DESCRIPTION

[0027] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] Current speech synthesis models usually use a vectorization method to assign an independent vector to each speaker (the vector is composed of multiple basic speech codes, and each basic speech code is used to represent the different acoustic parameter information of the corresponding speaker) to represent its speech feature information. However, when faced with massive data sets, such as speech data containing millions or even tens of millions of different speakers, assigning an independent vector to each speaker directly leads to an explosive growth in memory usage. In addition, due to the uneven distribution of data, some speakers have sufficient data, while others have seriously insufficient data, resulting in speakers with sufficient data having a large number of basic speech codes, and speakers with insufficient data having a small number of basic speech codes. Therefore, when performing speech synthesis, a large amount of resources may be allocated to speakers with sufficient data, while speakers with insufficient data are ignored. This not only increases the redundancy of the model and reduces the overall effect of speech synthesis, but also causes uneven utilization of basic speech codes.

[0029] For example, for a 2×10 7 The traditional method uses 2×10 7A separate vector is used to represent the speech feature information of each speaker, which takes up a lot of memory space. In addition, the coupling between different speakers is low, and the amount of speech data of some speakers is small, resulting in a large redundancy of the model.

[0030] To this end, the present invention provides a speech synthesis method. Figure 1 It is a flow chart of the speech synthesis method provided by the present invention, such as Figure 1 As shown, the method includes step 110 , step 120 , step 130 , and step 140 .

[0031] Step 110: extract text features of the text to be synthesized.

[0032] Here, the text to be synthesized refers to the text content that needs to be synthesized. The text to be synthesized can be any form of text, such as sentences, paragraphs, or complete articles, depending on the application scenario of speech synthesis. For example, in scenarios such as voice assistants, audiobook generation, and automatic voice notifications, the text to be synthesized can be user-entered instructions, the text content of a book, or notification information generated by the system.

[0033] In addition, text features refer to the semantic information, style information, sentiment information, etc. extracted from the text to be synthesized, which are used to represent the text to be synthesized. Among them, text features can include linguistic features (such as phonemes, syllables, parts of speech, etc.), semantic features, etc.

[0034] Exemplarily, when extracting text features of the text to be synthesized, the text features can be extracted from the text to be synthesized based on a machine learning model, such as extracting text features from the text to be synthesized based on a BERT model.

[0035] Step 120: Based on the identifier of the target speaker, search the speaker codebook for the target speech code that matches the target speaker. The speaker codebook stores speech codes of different speakers, where the speech code of each speaker is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0036] Specifically, the target speaker refers to the person for whom speech synthesis is required, and it can also be understood that when performing speech synthesis on the synthesized text, the generated synthesized speech needs to imitate the voice of the target person. The target speaker identifier is used to represent the identity of the target speaker, and the identifier can be represented by the target speaker's ID.

[0037] Usually, a mapping relationship between the identifiers of different speakers and the corresponding speech codes is established in the speaker codebook. Therefore, after determining the identifier of the target speaker, the target speech code matching the target speaker can be found in the speaker codebook based on the mapping relationship. The target speech code is used to characterize the speech feature information of the target speaker, such as pitch, speech speed, intonation, pronunciation habits, etc. The speaker codebook stores speech codes of different speakers, which are used to characterize the speech feature information of the corresponding speakers.

[0038] In addition, the speech characteristics of each speaker contain a large number of acoustic parameters and details, which need to be fully described by a series of basic speech codes. However, when constructing a speaker codebook, directly storing all the basic speech codes of each speaker to represent the speech characteristics of each speech will not only take up a huge storage space, but also increase the complexity of data processing and the delay of real-time synthesis.

[0039] Considering that some basic speech codes may contain repeated or similar information when describing the speech characteristics of a speaker, this information is not necessary for distinguishing different speakers or synthesizing high-quality speech. In this regard, some codes can be randomly selected from the basic speech codes corresponding to each speaker as the speech code of the speaker, thereby reducing the above redundant information to a certain extent, thereby reducing the storage and processing costs.

[0040] In addition, when constructing a speaker codebook, it is considered that some speakers may have a large number of basic speech codes, while other speakers may have only a small number of basic speech codes. This imbalance will not only affect the training effect of the speech synthesis model, but may also lead to uneven quality of synthesized speech, especially for speakers with less data. The embodiment of the present invention selects some codes from multiple basic speech codes of each speaker as the speech code of the corresponding speaker, thereby ensuring that each speaker has a relatively balanced basic speech code in the speaker codebook, thereby avoiding data redundancy.

[0041] Furthermore, considering that the speech characteristics of each speaker are unique, including specific acoustic parameters, intonation patterns, pronunciation habits and other key information. In order to ensure that these unique speech characteristics can be fully captured and accurately reproduced by the system, it is necessary to assign at least one basic speech code to each speaker as its speech code. When the total number of corresponding partial codes of all speakers is greater than or equal to the total number of speakers, it can be ensured that each speaker is assigned at least one basic speech code as a speech code, thereby achieving comprehensive data coverage.

[0042] Step 130: Based on the text features and the target speech coding, speech synthesis is performed on the text to be synthesized to obtain synthesized speech.

[0043] Specifically, text features are used to characterize the linguistic information contained in the text to be synthesized, and the target coding is used to characterize the speech features of the target speaker. The semantic content and expression style of the synthesized speech can be determined based on the text features. At the same time, the acoustic parameters of the synthesized speech can be adjusted based on the target speech coding to generate synthesized speech that conforms to the style of the target speaker.

[0044] Optionally, after obtaining the synthesized speech, the generated synthesized speech may be post-processed, including steps such as removing noise and optimizing sound quality, so as to improve the naturalness and fluency of the synthesized speech.

[0045] In the speech synthesis method provided by the embodiment of the present invention, the speech code of each speaker in the speaker codebook is a partial code selected from multiple basic speech codes of each speaker, so as to avoid directly storing all the basic speech codes of each speaker to represent the speech features of each speech, resulting in the problem of occupying a large storage space, and realizing the compression of the original speaker codebook. In addition, the embodiment of the present invention selects a partial code from multiple basic speech codes of each speaker as the speech code of the corresponding speaker, so as to ensure that each speaker has a relatively balanced basic speech code in the speaker codebook, so as to avoid data redundancy. Furthermore, the total number of partial codes corresponding to all speakers in the speaker codebook is greater than or equal to the total number of speakers, so as to ensure that each speaker is assigned at least one basic speech code as the speech code, thereby realizing the comprehensiveness of data coverage. On the basis of ensuring the accuracy of speech synthesis, the model redundancy is significantly reduced, and the utilization rate of the model parameters is improved.

[0046] Based on the above embodiment, the speech coding of each speaker is determined based on the following steps: Determine the size of the speaker codebook based on the total number of speakers; Based on the size of the speaker codebook, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

[0047] Specifically, the speaker codebook size is used to characterize the number of basic speech codes that can be stored. In order to ensure that the speaker codebook can accommodate the basic speech codes of all speakers, the embodiment of the present invention determines the speaker codebook size based on the total number of speakers. After determining the speaker codebook size, the upper limit of the basic speech codes that the speaker codebook can accommodate can be determined, that is, the sum of the basic speech codes of all speakers needs to be less than or equal to the upper limit of the basic speech codes that the speaker codebook can accommodate.

[0048] On the basis of ensuring that the sum of the basic speech codes of all speakers is less than or equal to the upper limit of the basic speech codes that can be accommodated in the speaker codebook, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

[0049] For example, each speaker includes n basic voice encodings, and m basic voice encodings can be selected from the n basic voice encodings of each speaker as their voice encodings (m < n). When selecting m basic voice encodings, clustering analysis can be performed based on the similarity between the basic voice encodings among the n basic voice encodings of the corresponding speaker, dividing the n basic voice encodings into m groups, and randomly selecting one basic voice encoding from each group as the voice encoding of the speaker.

[0050] Based on any of the above embodiments, selecting some encodings from the multiple basic voice encodings of each speaker as the voice encodings of each speaker includes: Based on the codebook sparsity factor and the speaker codebook size, determining the number of corresponding partial encodings for each speaker, and randomly selecting the corresponding number of partial encodings from the multiple basic voice encodings of each speaker as the voice encodings of each speaker; Or, Randomly sorting each speaker, and based on the binary representation corresponding to the sorting serial number of each speaker, selecting some encodings from the multiple basic voice encodings of each speaker as the voice encodings of each speaker.

[0051] Specifically, the codebook sparsity rate is used to characterize the proportion of the number of corresponding partial encodings of each speaker in the number of corresponding basic voice encodings. For example, each speaker includes n basic voice encodings, and m basic voice encodings are selected from the n basic voice encodings of each speaker as their voice encodings, then the codebook sparsity factor = m / n.

[0052] In addition, the speaker codebook size is used to characterize the number of basic voice encodings that the codebook can store for each speaker, that is, the speaker codebook size can be determined by the number of basic voice encodings of each speaker in the speaker codebook.

[0053] Based on this, according to the codebook sparsity factor and the speaker codebook size, the number of corresponding partial encodings for each speaker can be determined, and the corresponding number of partial encodings can be randomly selected from the multiple basic voice encodings of each speaker as the voice encodings of each speaker.

[0054] Suppose there are 3 million speakers, and n basic voice encodings with a dimension of 1024 are required for representation. The voice encoding of each speaker is represented by m basic voice encodings among the n basic voice encodings. Define the codebook sparsity factor r = m / n. Usually, the codebook sparsity factor is set to about 5%. Therefore, for a speaker codebook of size n×1024, the number of speakers that can be represented is as follows: For 3 million speakers, it is necessary to satisfy > 3000000, where n = 95 and m = 4 are selected here, and P(95, 4) = <3183545> 3000000 is calculated. The codebook sparsity factor r = m / n = 4 / 95 = 4.2% to ensure that each speaker has the same number of basic voice encodings for representation.

[0055] For example, for the voice data of 10 speakers, based on the method of the above embodiment, the size of the speaker codebook is n = 5, that is, each person in the speaker codebook includes 5 basic voice encodings, and the number of valid basic voice encodings m selected from the 5 basic voice encodings is 2. Table 1 is one of the voice encoding information lists. As shown in Table 1, each speaker includes 5 basic voice encodings, and "1" indicates the selected valid basic voice encoding. For example, the first speaker selects the first and second basic voice encodings as valid basic voice encodings, and the remaining digits are "0", that is, the basic voice encodings of the remaining digits are not included in the speaker codebook.

[0056] Table 1 In addition, as another optional method, each speaker can be randomly sorted, and based on the binary representation corresponding to the sorting serial number of each speaker, some encodings are selected from the multiple basic voice encodings of each speaker as the voice encoding of each speaker. Among them, it can be determined whether to select the corresponding basic voice encoding according to the value of each bit in the binary representation. For example, if a certain bit in the binary representation is "1", it means to select the basic voice encoding at the corresponding position, and if a certain bit in the binary representation is "0", it means not to select the basic voice encoding at the corresponding position.

[0057] Among them, the size of the codebook can be determined based on the number of bits of the binary representation. For example, if the number of bits of the binary representation is b, the size of the speaker codebook is b×1024, and the upper limit of the basic voice encodings that the corresponding speaker codebook can accommodate is . To achieve data full coverage, it is necessary to satisfy , where represents the number of speakers.

[0058] For example, if there are 3000000 speakers, then b = 22 can satisfy = <4194303> 3000000, so as to further compress the size of the speaker codebook, and the number of partial encodings selected by each speaker is different.

[0059] Table 2 is the second list of speech coding information. As shown in Table 2, the speaker codebook size b=4, that is, the valid basic speech coding of all speakers under the first position, the valid basic speech coding of all speakers under the second position, the valid basic speech coding of all speakers under the third position, and the valid basic speech coding of all speakers under the fourth position are stored through 4 speaker codebooks. The binary representation corresponding to the number "1" of the first speaker is "1000", that is, the first digit is "1", that is, the first basic speech coding is selected as the valid basic speech coding, and the remaining digits are "0", that is, the basic speech coding of the remaining digits is not included in the speaker codebook.

[0060] Table 2 In order to ensure the hit probability of the basic speech code in each speaker's codebook, the basic speech code can be randomly shuffled, and then randomly selected from the shuffled basic speech code. In addition, the effective speech code can be regularized to ensure that each effective speech code has an equal contribution in speech synthesis.

[0061] Based on any of the above embodiments, based on the text features and the target speech coding, performing speech synthesis on the text to be synthesized to obtain the synthesized speech includes: Based on the prior model, the text features and the target speech encoding are applied to generate prior features; Based on the speech synthesis model, text features, target speech coding and prior features are applied to perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0062] Specifically, prior features are intermediate features generated by a priori models based on text features and target speech encoding before speech synthesis, which can capture the inherent properties of speech (such as rhythm, intonation, pronunciation habits, etc.).

[0063] In addition, the prior features contain the association information between text features and target speech encoding (such as the correspondence between text content and speech features, the correspondence between emotional information and text content, the pronunciation relationship between adjacent words, etc.), so that the speech synthesis model can more accurately capture these features when generating speech, thereby improving the accuracy of speech synthesis.

[0064] The embodiment of the present invention can make full use of the prior features to enhance the effect of speech synthesis by first generating a priori features and then performing speech synthesis based on the prior features, text features and target speech coding. The prior features can capture the inherent properties of speech, enhance the accuracy of speech synthesis and improve the generalization ability of the model, thereby generating more natural and fluent synthesized speech that meets human auditory expectations.

[0065] Based on any of the above embodiments, the speech synthesis model is trained by combining the prior model and the posterior model. The speech synthesis model is trained based on the following steps: Extracting sample text features of the sample synthesized text, and searching the speaker codebook for a first sample speech code that matches the sample speaker based on an identifier of the sample speaker; Based on the initial model of the prior model, applying the sample text feature and the first sample speech coding to obtain the first sample prior feature; Based on the initial model of the posterior model, feature restoration is performed on the synthetic speech features of the synthetic speech label to obtain sample posterior features, where the sample posterior features have the same dimension as the first sample prior features; Based on the initial model of the speech synthesis model, applying the first sample prior features, the sample text features and the first sample speech encoding, obtain the first sample synthesized speech; Based on the difference between the first sample prior features and the sample posterior features, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model and the posterior model.

[0066] Specifically, the prior model is used to capture the correlation information between the sample text features and the first sample speech code, as well as the inherent speech properties in the first sample speech code, so as to provide reliable prior knowledge for speech synthesis and guide the speech synthesis model to perform speech synthesis on the sample synthetic text. The posterior model is used to restore the features of the synthesized speech to obtain sample posterior features with the same dimension as the first sample prior features. The synthesized speech features can be found in a preset speech codebook based on the identification of the sample speaker.

[0067] The difference between the first sample prior feature and the sample posterior feature reflects the prediction accuracy of the prior model for the prior feature. The smaller the difference between the two, the smaller the difference between the first sample prior feature predicted by the prior model and the actual feature (i.e., the sample posterior feature), and thus the prior model can accurately predict the prior feature. With the training goal of minimizing the difference between the first sample prior feature and the sample posterior feature, the prior model can be trained to more accurately capture the correlation information between text and speech, as well as the inherent properties of speech, and thus more accurately predict the prior feature.

[0068] In addition, the difference between the first sample synthesized speech and the synthesized speech label is used to characterize the difference between the synthesized speech output by the speech synthesis model and the actual speech. By minimizing the difference between the first sample synthesized speech and the synthesized speech label as the training goal, the speech synthesis model can be trained to more accurately generate synthesized speech that matches the actual speech, thereby improving the naturalness and authenticity of the synthesized speech.

[0069] Figure 2 : is a schematic diagram of the training process of the speech synthesis model provided by the present invention, such as Figure 2 As shown, in the preset speech codebook (speech_codec), the synthesized speech features are found based on the sample speaker's identifier (speech_codec), and the synthesized speech features are input into the prior-posteriori feature extraction model (deep_attention) to obtain the sample posterior features.

[0070] At the same time, based on the sample speaker identifier (spk_id), the first sample speech encoding matching the sample speaker is searched in the speaker codebook (spk_codebook), and the first sample speech encoding is input into the prior and posterior feature extraction model (deep_attention), and the sample text feature (text_encode_input) of the sample synthesized text is input into the prior and posterior feature extraction model (deep_attention) to obtain the first sample prior feature.

[0071] Inputting the first sample priori feature, the sample text feature and the first sample speech code into the speech synthesis model to obtain the first sample synthesized speech; Based on the difference between the first sample prior features and the sample posterior features, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the prior and posterior feature extraction model (deep_attention) and the speech synthesis model are jointly updated to obtain a trained speech synthesis model.

[0072] Figure 3 It is a structural diagram of the a priori and a posteriori feature extraction model provided by the present invention, such as Figure 2 As shown, the prior and posterior feature extraction model (deep_attention) includes a prior model and a posterior model. The prior model includes a prior feature extraction layer (map_attention), a normalization layer (layernorm), a FFN network, and a normalization layer (layernorm) in sequence. After the sample text features and the first sample speech code (y_feature) are input into the prior feature extraction layer, they pass through the normalization layer, the FFN network, and the normalization layer in sequence to obtain the first sample prior feature (y_output).

[0073] The posterior model includes the prior feature extraction layer (map_attention), the normalization layer (layernorm), the FFN network, the normalization layer (layernorm), the first posterior feature extraction layer (map_mean_layer) and the second posterior feature extraction layer (map_var_layer). The synthesized speech feature (x_feature) is input into the prior feature extraction layer (map_attention), and passes through the normalization layer (layernorm), the FFN network, the normalization layer (layernorm), the first posterior feature extraction layer (map_mean_layer) and the second posterior feature extraction layer (map_var_layer) in turn to obtain the sample posterior feature (x_output), and the loss value (Loss) is determined based on the first sample prior feature (y_output) and the sample posterior feature (x_output).

[0074] Figure 4 is a schematic diagram of the structure of the prior feature extraction layer provided by the present invention, such as Figure 4 As shown, the prior feature extraction layer (map_attention) includes a key convolution network (conv1d_k), a value convolution network (conv1d_v), a matmul module, a drop layer, and a softmax layer. The prior feature extraction process of this layer is as follows: the input feature (input_feature) is sent to the key convolution network (conv1d_k) and the value convolution network (conv1d_v) respectively to obtain the corresponding convolution features, and then the two convolution features are multiplied to obtain a matrix feature of a fixed size to obtain the codebook feature. The codebook feature is a fixed-dimensional feature and has nothing to do with the time dimension feature. For example, assuming that the input feature dimension is B×T×C (where B represents the size of the batch data, T is the time dimension feature, and C is the extracted fixed-dimensional feature), the key feature and value feature can be obtained respectively through the key convolution network (conv1d_k) and the value convolution network (conv1d_v), and the corresponding feature dimensions are B×T×C. Then the matmul module is input for matrix multiplication to obtain a fixed codebook feature with a feature dimension of B×C×C. Finally, the fixed codebook feature is processed by the drop layer and the softmax layer to obtain the final codebook feature output_feature.

[0075] Figure 5 It is a schematic diagram of the structure of the FFN network provided by the present invention, such as Figure 5As shown in the figure, the execution process of the FFN network is to first input the input feature (input_feature) into the convolution upsampling network (conv1d_upsample) for dimension upsampling to upsample the C dimension to 4×C, then process the upscaled feature through the activation function, and then input it into the convolution downsampling network (conv1d_downsample) to restore the feature of dimension 4×C to the feature of dimension C, and finally output the restored feature.

[0076] Figure 6 is a schematic diagram of the structure of the posterior feature extraction layer provided by the present invention, such as Figure 6 As shown in the figure, the posterior feature extraction layer (map_mean_layer and map_var_layer have the same structure) first inputs the input feature (input_feature) into the linear upsampling network (linear_upsample) to upsample the feature with dimension C to 4×C, then inputs the upsampled feature into the relu activation function for feature transformation, and finally inputs the transformed feature into the linear downsampling network (linear_downsample) for downsampling to obtain the feature with dimension C, which is then output.

[0077] Based on any of the above embodiments, based on the difference between the first sample prior feature and the sample posterior feature, and the difference between the first sample synthesized speech and the synthesized speech label, jointly updating the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model to obtain the speech synthesis model, the prior model, and the posterior model, and then further including: Extracting sample speech features of the sample speaker, and searching for a second sample speech code matching the sample speech features in the speaker codebook based on the sample speech features; Based on the prior model, applying the second sample speech coding and the sample text features to obtain the second sample prior features; Based on the speech synthesis model, applying the sample text feature, the second sample prior feature and the second sample speech encoding input to obtain the second sample synthesized speech; Based on the difference between the second sample prior feature and the sample posterior feature, and the difference between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0078] Specifically, based on the difference between the first sample prior feature and the sample posterior feature, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model, and the posterior model. Then, the parameters of the speech synthesis model and the prior model can be fine-tuned so that the speech synthesis model and the prior model can generate the synthesized speech of a specific sample speaker. It should be noted that in order to retain the bottom model compression information of the above-mentioned big data training as much as possible, after obtaining the speech synthesis model, the prior model, and the posterior model, the embodiments of the present invention fix the parameters of the posterior model and fine-tune the parameters of the speech synthesis model and the prior model.

[0079] Figure 7 FIG. 1 is a schematic diagram of the fine-tuning process of the speech synthesis model provided by the present invention. Figure 7 As shown in the figure, the fine-tuning process is as follows: extract the sample speech features (spk_embed) of the sample speaker, and based on the sample speech features, find the second sample speech encoding that matches the sample speech features in the speaker codebook (spk_codebook), input the second sample speech encoding (text_encode_input) and the sample text features into the prior and posterior feature extraction model (deep_attention) to obtain the second sample prior features.

[0080] In the preset speech codebook (speech_codec), the synthesized speech features are searched based on the sample speaker's identifier (speech_codec), and the synthesized speech features are input into the prior and posterior feature extraction model (deep_attention) to obtain the sample posterior features.

[0081] The sample text features, the second sample prior features and the second sample speech code are input into the speech synthesis model to obtain the second sample synthesized speech (speech_codec). Based on the difference between the second sample prior features and the sample posterior features, and the difference between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0082] Based on any of the above embodiments, based on the speech synthesis model, applying text features, target speech coding and prior features, performing speech synthesis on the text to be synthesized to obtain synthesized speech, including: Constructing a speech synthesis prompt text, where the speech synthesis prompt text is used to indicate the speech synthesis requirements of the text to be synthesized; Based on the speech synthesis model, the speech synthesis prompt text, text features, target speech coding and prior features are applied to perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0083] Specifically, different texts may require different voice styles, speaking speeds, tones, etc. For example, a news report may require a formal, steady tone, while a children's story may require a friendly, lively tone. The speech synthesis prompt text can clearly indicate these requirements, enabling the speech synthesis model to generate corresponding speech according to different requirements.

[0084] The embodiments of the present invention can not only improve the accuracy and naturalness of speech synthesis by combining the speech synthesis requirements for indicating the text to be synthesized, but also meet diverse speech synthesis needs, thereby improving the user experience.

[0085] like Figure 2 and Figure 7 As shown, during the model training and fine-tuning process, the speech synthesis prompt text (prompt_speech_codec) can also be input into the speech synthesis model to guide the speech synthesis model to generate synthesized speech that meets the requirements.

[0086] The speech synthesis device provided by the present invention is described below. The speech synthesis device described below and the speech synthesis method described above can be referred to each other.

[0087] Based on any of the above embodiments, Figure 8 is a structural schematic diagram of the speech synthesis device provided by the present invention, such as Figure 8 As shown, the device comprises: An extraction unit 810, used to extract text features of the text to be synthesized; A search unit 820 is used to search a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; A synthesis unit 830 is used to perform speech synthesis on the text to be synthesized based on the text features and the target speech coding to obtain a synthesized speech; The speaker codebook stores speech codes of different speakers. The speech code of each speaker is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0088] Based on any of the above embodiments, the speech coding of each speaker is determined based on the following steps: Determine the size of the speaker codebook based on the total number of speakers; Based on the size of the speaker codebook, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

[0089] Based on any of the above embodiments, based on the size of the speaker codebook, selecting part of the codes from the multiple basic speech codes of each speaker as the speech code of each speaker includes: Based on the codebook sparsity factor and the speaker codebook size, determine the number of partial codes corresponding to each speaker, and randomly select a corresponding number of partial codes from multiple basic speech codes of each speaker as the speech code of each speaker; or, The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

[0090] Based on any of the above embodiments, based on the text features and the target speech coding, performing speech synthesis on the text to be synthesized to obtain the synthesized speech includes: Based on the prior model, the text features and the target speech encoding are applied to generate prior features; Based on the speech synthesis model, text features, target speech coding and prior features are applied to perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0091] Based on any of the above embodiments, the speech synthesis model is trained by combining the prior model and the posterior model. The speech synthesis model is trained based on the following steps: Extracting sample text features of the sample synthesized text, and searching for a first sample speech code matching the sample speaker in a speaker codebook based on an identifier of the sample speaker; Based on the initial model of the prior model, applying the sample text feature and the first sample speech coding to obtain the first sample prior feature; Based on the initial model of the posterior model, feature restoration is performed on the synthetic speech features of the synthetic speech label to obtain sample posterior features, where the sample posterior features have the same dimension as the first sample prior features; Based on the initial model of the speech synthesis model, applying the first sample prior features, the sample text features and the first sample speech encoding, obtain the first sample synthesized speech; Based on the difference between the first sample prior features and the sample posterior features, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model and the posterior model.

[0092] Based on any of the above embodiments, based on the difference between the first sample prior feature and the sample posterior feature, and the difference between the first sample synthesized speech and the synthesized speech label, jointly updating the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model to obtain the speech synthesis model, the prior model, and the posterior model, and then further including: Extracting sample speech features of the sample speaker, and searching for a second sample speech code matching the sample speech features in the speaker codebook based on the sample speech features; Based on the prior model, applying the second sample speech coding and the sample text features to obtain the second sample prior features; Based on the speech synthesis model, applying the sample text feature, the second sample prior feature and the second sample speech encoding input to obtain the second sample synthesized speech; Based on the difference between the second sample prior feature and the sample posterior feature, and the difference between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

[0093] Based on any of the above embodiments, based on the speech synthesis model, applying text features, target speech coding and prior features, performing speech synthesis on the text to be synthesized to obtain synthesized speech, including: Constructing a speech synthesis prompt text, where the speech synthesis prompt text is used to indicate the speech synthesis requirements of the text to be synthesized; Based on the speech synthesis model, the speech synthesis prompt text, text features, target speech coding and prior features are applied to perform speech synthesis on the text to be synthesized to obtain synthesized speech.

[0094] Fig. 9 is a schematic diagram of the structure of the electronic device provided by the present invention, such as Fig. 9 As shown, the electronic device may include: a processor 910, a communication interface 920, a memory 930 and a communication bus 940, wherein the processor 910, the communication interface 920 and the memory 930 communicate with each other through the communication bus 940. The processor 910 may call the logic instructions in the memory 930 to execute the speech synthesis method, which includes: extracting text features of the text to be synthesized; based on the identifier of the target speaker, searching the speaker codebook for the target speech code matching the target speaker; based on the text features and the target speech code, performing speech synthesis on the text to be synthesized to obtain synthesized speech; the speaker codebook stores speech codes of different speakers, and the speech code of each speaker is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0095] In addition, the logic instructions in the above-mentioned memory 930 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0096] On the other hand, the present invention also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the speech synthesis method provided by the above methods, which method includes: extracting text features of the text to be synthesized; based on the identification of the target speaker, searching for a target speech code matching the target speaker in a speaker codebook; based on the text features and the target speech code, performing speech synthesis on the text to be synthesized to obtain synthesized speech; the speaker codebook stores speech codes of different speakers, the speech code of each speaker is a partial code selected from multiple basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0097] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the speech synthesis method provided by the above-mentioned methods, the method comprising: extracting text features of a text to be synthesized; based on an identifier of a target speaker, searching a speaker codebook for a target speech code that matches the target speaker; based on the text features and the target speech code, performing speech synthesis on the text to be synthesized to obtain synthesized speech; the speaker codebook stores speech codes of different speakers, the speech code of each speaker being a partial code selected from a plurality of basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

[0098] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0099] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0100] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A speech synthesis method, characterized in that: include: Extracting text features of the text to be synthesized; Based on the identifier of the target speaker, searching the speaker codebook for a target speech code that matches the target speaker; Based on the text features and the target speech coding, performing speech synthesis on the text to be synthesized to obtain synthesized speech; The speaker codebook stores speech codes of different speakers, wherein the speech code of each speaker is a partial code selected from a plurality of basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

2. The speech synthesis method according to claim 1, characterized in that: The speech code for each speaker is determined based on the following steps: Determining a speaker codebook size based on the total number of speakers; Based on the speaker codebook size, some codes are selected from a plurality of basic speech codes of each speaker as the speech code of each speaker.

3. The speech synthesis method according to claim 2, characterized in that: The selecting part of the codes from the multiple basic speech codes of each speaker as the speech code of each speaker based on the size of the speaker codebook includes: Determining the number of partial codes corresponding to each speaker based on the codebook sparsity factor and the speaker codebook size, and randomly selecting a corresponding number of partial codes from a plurality of basic speech codes of each speaker as the speech code of each speaker; or, The speakers are randomly sorted, and based on the binary representation corresponding to the sorting number of each speaker, some codes are selected from the multiple basic speech codes of each speaker as the speech code of each speaker.

4. The speech synthesis method according to any one of claims 1 to 3, characterized in that: The step of performing speech synthesis on the text to be synthesized based on the text features and the target speech coding to obtain synthesized speech includes: Based on the prior model, applying the text features and the target speech coding to generate prior features; Based on the speech synthesis model, the text features, the target speech coding and the prior features are applied to perform speech synthesis on the text to be synthesized to obtain the synthesized speech.

5. The speech synthesis method according to claim 4, characterized in that: The speech synthesis model is trained by combining the prior model and the posterior model. The speech synthesis model is trained based on the following steps: Extracting sample text features of the sample synthesized text, and searching the speaker codebook for a first sample speech code matching the sample speaker based on an identifier of the sample speaker; Based on the initial model of the prior model, applying the sample text feature and the first sample speech coding to obtain a first sample prior feature; Based on the initial model of the posterior model, feature restoration is performed on the synthesized speech features of the synthesized speech label to obtain sample posterior features, where the sample posterior features have the same dimension as the first sample a priori features; Based on the initial model of the speech synthesis model, applying the first sample prior features, the sample text features and the first sample speech coding to obtain a first sample synthesized speech; Based on the difference between the first sample prior features and the sample posterior features, and the difference between the first sample synthesized speech and the synthesized speech label, the parameters of the initial model of the prior model, the initial model of the posterior model and the initial model of the speech synthesis model are jointly updated to obtain the speech synthesis model, the prior model and the posterior model.

6. The speech synthesis method according to claim 5, characterized in that: The method further comprises: jointly updating the parameters of the initial model of the prior model, the initial model of the posterior model, and the initial model of the speech synthesis model based on the difference between the first sample prior feature and the sample posterior feature, and the difference between the first sample synthesized speech and the synthesized speech label, to obtain the speech synthesis model, the prior model, and the posterior model, and then further comprising: Extracting a sample speech feature of a sample speaker, and searching the speaker codebook for a second sample speech code that matches the sample speech feature based on the sample speech feature; Based on the prior model, applying the second sample speech coding and the sample text feature to obtain a second sample prior feature; Based on the speech synthesis model, applying the sample text feature, the second sample prior feature and the second sample speech coding input to the to obtain a second sample synthesized speech; Based on the difference between the second sample prior feature and the sample posterior feature, and the difference between the second sample synthesized speech and the synthesized speech label, the parameters of the speech synthesis model and the prior model are fine-tuned.

7. The speech synthesis method according to claim 4, characterized in that: The method of performing speech synthesis on the text to be synthesized by applying the text feature, the target speech coding and the prior feature based on the speech synthesis model to obtain the synthesized speech includes: Constructing a speech synthesis prompt text, wherein the speech synthesis prompt text is used to indicate the speech synthesis requirement of the text to be synthesized; Based on the speech synthesis model, the speech synthesis prompt text, the text features, the target speech coding and the prior features are applied to perform speech synthesis on the text to be synthesized to obtain the synthesized speech.

8. A speech synthesis device, characterized in that: include: An extraction unit, used for extracting text features of the text to be synthesized; A search unit, configured to search a target speech code matching the target speaker in a speaker codebook based on the identifier of the target speaker; A synthesis unit, configured to perform speech synthesis on the text to be synthesized based on the text features and the target speech coding to obtain synthesized speech; The speaker codebook stores speech codes of different speakers, wherein the speech code of each speaker is a partial code selected from a plurality of basic speech codes of each speaker, and the total number of partial codes corresponding to all speakers is greater than or equal to the total number of speakers.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that: When the processor executes the computer program, the speech synthesis method according to any one of claims 1 to 7 is implemented.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the speech synthesis method according to any one of claims 1 to 7 is implemented.

Images