Speech synthesis method and device based on focus information, equipment and storage medium

Abstract

The application provides a speech synthesis method and device based on focus information, equipment and a storage medium. The method learns focus information in a text according to a text sample and a corresponding sample focus classification label, and trains a focus prediction model. Then, the focus prediction model is used to predict target focus classification information of emphasized text for representing to-be-synthesized text. According to the target focus classification information, a target synthesized speech with rhythm changes corresponding to the to-be-synthesized text is controlled. Thus, according to the target focus classification information, the focus text to be emphasized in the to-be-synthesized text is highlighted, the focus text to be emphasized and non-focus text produce rhythm changes in the target synthesized speech, the rhythm naturalness of the synthesized speech is improved, and the technical problem of low rhythm naturalness of the synthesized speech is solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method and device based on focus information, an equipment and a computer readable storage medium. BACKGROUND

[0002] Speech synthesis technology (Text To Speech, TTS) is a branch of artificial intelligence research, which is used to convert text information into speech information. Speech synthesis technology is divided into a speech synthesis front end and a speech synthesis back end. The front end is used to convert a text sequence into a corresponding phoneme sequence, which includes prosody, multi-syllable words and stress information. The back end is used to take the front-end phoneme sequence as an input acoustic model, and convert the front-end phoneme sequence into speech through the acoustic model. At present, the speech generated by the speech synthesis technology is generally of average prosody, and cannot emphasize the parts that need to be emphasized in a sentence, resulting in a strong mechanical feeling of the generated speech and low prosody naturalness. Therefore, how to solve the low prosody naturalness of the current synthesized speech has become a technical problem to be solved. SUMMARY

[0003] The present application provides a speech synthesis method and device based on focus information, an equipment and a storage medium, aiming at solving the technical problem of low prosody naturalness of the current synthesized speech.

[0004] In a first aspect, the present application provides a speech synthesis method based on focus information, which comprises:

[0005] Generating target focus classification information of a text to be synthesized based on a focus prediction model, wherein the focus prediction model is trained and generated according to a text sample and a sample focus classification label corresponding to the text sample;

[0006] Obtaining a target prosody label of the text to be synthesized, and generating a target local prosody representation of the text to be synthesized according to the text to be synthesized, the target focus classification information and the target prosody label;

[0007] Generating target synthesized speech of the text to be synthesized according to the target local prosody representation.

[0008] In a second aspect, the present application provides a speech synthesis device based on focus information, which comprises:

[0009] A focus information prediction module is configured to generate target focus classification information of a text to be synthesized based on a focus prediction model, wherein the focus prediction model is trained and generated according to a text sample and a sample focus classification label corresponding to the text sample;

[0010] a prosody representation generation module, configured to obtain a target prosody label of the text to be synthesized, and generate a target local prosody representation of the text to be synthesized according to the text to be synthesized, the target focus level information, and the target prosody label;

[0011] a target speech synthesis module, configured to generate a target synthesized speech of the text to be synthesized according to the target local prosody representation.

[0012] In a third aspect, an embodiment of the present application provides a speech synthesis device based on focus information, which comprises a processor, a memory, and a speech synthesis program stored in the memory and executable by the processor, wherein the speech synthesis program, when executed by the processor, implements the steps of the speech synthesis method described above.

[0013] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, wherein the computer program, when executed by a processor, implements the steps of the speech synthesis method described above.

[0014] The present application provides a speech synthesis method, device, equipment and storage medium based on focus information. The method generates target focus level information of text to be synthesized based on a focus prediction model, wherein the focus prediction model is trained and generated according to a text sample and a sample focus level label corresponding to the text sample. The method obtains a target prosody label of the text to be synthesized, and generates a target local prosody representation of the text to be synthesized according to the text to be synthesized, the target focus level information, and the target prosody label. The method generates a target synthesized speech of the text to be synthesized according to the target local prosody representation. Through the above method, the focus information in the text is learned according to the text sample and the corresponding sample focus level label, and the focus prediction model is trained and generated. Then, the target focus level information of the text to be emphasized is predicted by the focus prediction model. The target synthesized speech with prosody changes corresponding to the text to be synthesized is controlled according to the target focus level information. Thus, the focus text to be emphasized is highlighted according to the target focus level information, and the focus text to be emphasized and the non-focus text produce prosody changes in the target synthesized speech, which improves the prosody naturalness of the synthesized speech and solves the technical problem of low prosody naturalness of the synthesized speech.

[0015] It should be understood that the above general description and the following detailed description are only exemplary and explanatory, and cannot limit the disclosure of the embodiments of the present application. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 is a hardware structure schematic diagram of the voice synthesis device based on focus information provided by the embodiments of the present application;

[0018] Figure 2 is a flowchart of the first embodiment of the voice synthesis method based on focus information provided by the embodiments of the present application;

[0019] Figure 3 is a flowchart of the second embodiment of the voice synthesis method based on focus information provided by the embodiments of the present application;

[0020] Figure 4 is a model training schematic diagram of the focus prediction model provided by the embodiments of the present application;

[0021] Figure 5 is a flowchart of the third embodiment of the voice synthesis method based on focus information provided by the embodiments of the present application;

[0022] Figure 6 is a functional module schematic diagram of the voice synthesis device based on focus information provided by the embodiments of the present application. DETAILED DESCRIPTION

[0023] The technical solutions of the embodiments of the present application will be described clearly and completely in the following with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0024] The flowchart shown in the drawings is only an example description, not necessarily including all contents and operations / steps, and not necessarily executed in the described order. For example, some operations / steps can be decomposed, combined or partially merged, so the actual execution order can be changed according to the actual situation.

[0025] The embodiments of the present application will be described in detail below with reference to the drawings. In the case of no conflict, the embodiments described below and the features in the embodiments can be combined with each other.

[0026] Reference Figure 1 , Figure 1A hardware structure diagram of the voice synthesis device based on focus information according to an embodiment of the present application is shown in FIG. 1. In the embodiment of the present application, the voice synthesis device based on focus information can include a processor 1001 (for example, a CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to realize the connection and communication among the components; the user interface 1003 can include a display screen (Display) and an input unit such as a keyboard (Keyboard); the network interface 1004 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface); the memory 1005 can be a high-speed RAM memory, or a stable memory (non-volatile memory) such as a disk memory, and the memory 1005 can optionally be a storage device independent of the aforementioned processor 1001.

[0027] Those skilled in the art can understand that the hardware structure shown in FIG. 1 does not constitute a limitation on the voice synthesis device based on focus information, and can include more or fewer components than those shown in the figure, or combine certain components, or arrange different components. Figure 1

[0028] Referring to FIG. 1, Figure 1 , Figure 1 The memory 1005 in FIG. 1 as a computer readable storage medium can include an operating system, a network communication module, and a voice synthesis program based on focus information.

[0029] In FIG. 1, the network communication module is mainly used to connect to a server and communicate data with the server; and the processor 1001 can call the voice synthesis program based on focus information stored in the memory 1005 and execute the voice synthesis method based on focus information provided by the embodiment of the present application. Figure 1

[0030] Referring to FIG. 1, Figure 2 , Figure 2 A flowchart of the voice synthesis method based on focus information according to a first embodiment of the present application is shown in FIG. 2.

[0031] As shown in FIG. 2, the voice synthesis method based on focus information can be applied to, but not limited to, virtual image, intelligent interaction, intelligent customer service, news broadcast, novel audio reading, and intelligent vehicle application scenarios, and the voice synthesis method based on focus information includes steps S100 to S300. Figure 2

[0032] S100, generating target focus hierarchical information of a text to be synthesized based on a focus prediction model, wherein the focus prediction model is trained and generated according to a text sample and a sample focus hierarchical label corresponding to the text sample; ​​​

[0033] Currently, end-to-end speech synthesis acoustic models, such as autoregressive Transformer models, Tacotron models, Fast Speech models, and non-autoregressive paralleltts models, etc., encode the trained text as input at the encoder end, and then use the training method of teacher forcing of the recurrent neural network model at the decoder end to decode the output acoustic features. However, the above models can only learn the average prosody style in the training data set, and the prosody of the synthesized speech by the above models is relatively smooth, without highlighting the prosody changes that need to be emphasized, and cannot produce prosodic fluctuations, nor can it highlight the focus part of the speech like a real user (such as increasing the volume or tone length, etc.). Thus, the naturalness of the synthesized speech is low. In the present embodiment, a speech synthesis method based on focus information is provided, which predicts target focus classification information for indicating the text that needs to be emphasized in the to-be-synthesized text through a focus prediction model. And according to the target focus classification information, the target synthesized speech with prosodic changes corresponding to the to-be-synthesized text is controlled. Thus, according to the target focus classification information, the focus text that needs to be emphasized in the to-be-synthesized text is highlighted, so that the focus text that needs to be emphasized and the non-focus text produce prosodic changes in the target synthesized speech, and the prosodic naturalness of the synthesized speech is improved.

[0034] Specifically, the target focus classification information defines the focus position (such as the first word, the fifth word, etc. in the to-be-synthesized text) and the focus intensity information (such as strong, medium, and weak) in the to-be-synthesized text. Through the trained focus prediction model, the words that need to be emphasized in the to-be-synthesized text are identified, and the target focus classification information corresponding to the to-be-synthesized text is generated according to the identification result.

[0035] It can be understood that the sample focus classification label corresponding to the text sample used for training the focus prediction model can be manually set or extracted from the sample local prosody representation of the speech sample corresponding to the text sample according to the focus extraction model. If there is no speech sample corresponding to the text sample, the focus position of the text sample can be manually specified, the text with the specified focus position is added with a focus intensity label, and the focus prediction model is trained according to the text sample and the focus label. If there is a speech sample corresponding to the text sample, the sample focus classification information extracted from the sample local prosody representation of the speech sample by the focus extraction model can be used as a target to train the focus prediction model. The sample local prosody representation is to remove the speaker information and the text information, and only keep the information of the prosodic fluctuation, the fundamental frequency and the duration of the speech, which is used to represent the word-level prosodic changes within a sentence in the speech sample. The focus classification information in the local prosody representation is the sample position in the speech sample that needs to be read or lengthened for emphasis, that is, the focus position, and the focus intensity of the focus position sample, the focus frequency in the language sample and other information, that is, the focus classification information.

[0036] S200, obtaining a target prosody label of the text to be synthesized, and generating a target local prosody representation of the text to be synthesized according to the text to be synthesized, the target focus classification information and the target prosody label;

[0037] In this embodiment, the target focus classification information of the text to be synthesized is combined with the target prosody label of the text to be synthesized to input the prosody prediction model, and a target local prosody representation for representing the prosodic changes and the focus classification information of the text to be synthesized is generated.

[0038] S300, generating a target synthesized speech of the text to be synthesized according to the target local prosody representation.

[0039] In this embodiment, according to the target local prosody representation, the speech synthesis model can generate a target synthesized speech including the prosodic changes (different intensity focus categories) of the text to be synthesized.

[0040] Specifically, a method of unsupervised network and adversarial training is adopted to obtain a local prosody representation of a speech training sample, and speaker information, text information and prosodic change information (pitch fluctuation and pronunciation duration, etc.) in the speech are decoupled. That is, a uniform prosody representation representing prosodic change information of each word in the speech training sample is extracted from the speech training sample by an unsupervised network such as a variational autoencoder (VAE) or an autoencoder (AE). And mutual information constraint is performed according to the extracted local prosody representation and adversarial training to train a local prosody representation synthesis model. The above local prosody representation synthesis model can decouple the speaker information, the text information and the local prosody representation, extract only the prosodic change information including the pitch fluctuation and the pronunciation duration, and then input the local prosody representation into a speech synthesis model to obtain the target synthesized speech. Thus, by the focus prediction model, the focus level information corresponding to each word of the text to be synthesized is predicted to represent the focus intensity of each word to be synthesized, and the synthesized speech composed of words with different pitch fluctuations and different prosodic changes. The pitch of the word at the focus position in the synthesized speech is raised, and the pronunciation duration of the word at the focus position is lengthened, thereby emphasizing the word at the focus position of the text to be synthesized.

[0041] The embodiment of the present application provides a speech synthesis method, device and equipment based on focus information and a storage medium. The method is based on a focus prediction model to generate target focus level information of text to be synthesized. The focus prediction model is trained according to a text sample and a sample focus level label corresponding to the text sample. The target prosody label of the text to be synthesized is obtained, and the target local prosody representation of the text to be synthesized is generated according to the text to be synthesized, the target focus level information and the target prosody label. The target synthesized speech of the text to be synthesized is generated according to the target local prosody representation. According to the above method, the focus information in the text is learned according to the text sample and the corresponding sample focus level label, and the focus prediction model is trained. Then, the target focus level information of the text to be emphasized is predicted by the focus prediction model. The target synthesized speech with prosodic changes corresponding to the text to be synthesized is controlled according to the target focus level information. Thus, the focus text to be emphasized is highlighted according to the target focus level information, and the focus text to be emphasized and the non-focus text produce prosodic changes in the target synthesized speech, which improves the prosodic naturalness of the synthesized speech and solves the technical problem of low prosodic naturalness of the synthesized speech.

[0042] Please refer to Figure 3 , Figure 3 is a flowchart of the second embodiment of the speech synthesis method based on focus information. The speech synthesis method based on focus information further includes steps S400 to S600 before step S100.

[0043] S400, obtaining a speech sample corresponding to the text sample, and generating a sample prosodic representation of the speech sample according to a prosodic representation generation model;

[0044] S500, extracting sample focus grading information from the sample prosody representation as the sample focus grading label;

[0045] S600: Training and generating the focus prediction model based on the text samples and the sample focus grading labels.

[0046] In this embodiment, manually specifying the focus position of a text sample not only increases the training cost of the focus prediction model but also reduces its training efficiency. To reduce training cost and improve training efficiency, during focus prediction model training, sample focus classification information is extracted from the local prosodic representation of the speech sample corresponding to the text sample, and this sample focus classification information is used as the sample focus classification label for the text sample.

[0047] The local prosodic representation represents prosodic variation information including fundamental frequency fluctuation, fundamental frequency trend and duration information. Therefore, focus grading information for representing word-level prosodic variation information can be extracted from the local prosodic representation for representing character-level prosodic variation information.

[0048] like Figure 4 As shown, a text sample and a speech sample corresponding to the text sample are set, and a sample prosodic representation corresponding to the speech sample is generated based on the currently existing prosodic representation generation model. Therefore, the sample focus grading information is first extracted from the sample prosodic representation by the focus extraction model, and the extracted sample focus grading information is used as the target (i.e., the sample focus grading label corresponding to the text sample); then the text sample and the sample focus grading label are input into the unsupervised language model to learn the focus grading information in the text sample, and train and generate a focus prediction model. Among them, the above-mentioned focus prediction model can generate the target focus grading information corresponding to the text to be synthesized that does not include the speech text based on the learned focus control features.

[0049] Furthermore, the step of extracting sample focus classification information from the sample prosody representation specifically includes:

[0050] encoding the sample prosody representation into a continuous representation vector through a focus extraction model;

[0051] According to the distance between each focus category vector in the preset code table and the continuous characterization vector, the continuous characterization vector is quantized into a corresponding discrete characterization vector for representing the focus category as the sample focus grading information.

[0052] In this embodiment, the focus extraction model is used to encode the sample prosody representation into a continuous representation vector corresponding to the sample prosody representation, and then a neural discrete representation learning generation model VQVAE is used to quantize the continuous representation vector into a hierarchical discrete representation vector, so as to map the local prosody representation at the word level to the focus hierarchical information at the word level. The discrete representation vector is used to represent the sample focus hierarchical information. The sample focus hierarchical information is used to increase the fundamental frequency or the pronunciation duration of the sample corresponding to the focus position in the text sample. The local prosody representation is obtained by the encoding encoder end to obtain the continuous representation vector z c (x), the distance between z c (x) and each focus category e i in the codebook of the VQVAE is calculated by distance calculation, and the distance change is represented by the distance. According to the change, the closest e i (i.e., the e i with the smallest distance) is determined in each e i of the codebook. According to the e c closest to z i (x), z c (x) is mapped to the discrete representation vector z d (x).

[0053] In order to prevent the continuous representation vector z c (x) output by the encoder end from frequently jumping between e i in the codebook, that is, the closest e i cannot be determined, when updating the loss function Loss, the encoder end and the decoder end are updated by the updated loss, the encoder is trained by the commitment loss, and each focus category in the codebook is represented by the codebook loss to make the closest z c (x) close to each other.

[0054] Further, the training process of the focus prediction model is as follows:

[0055] A focus prediction model is constructed by taking a text sample as input to learn the part that needs to be emphasized in a speech sample from the text sample. Specifically, the text sample is encoded by an unsupervised language model (not limited to a BERT language model) and fused with the pronunciation phoneme information in the speech sample corresponding to the text sample. In order to prevent the text sample and the pronunciation phoneme information corresponding to the text sample from being misaligned, i.e., the text sample and the pronunciation phoneme information corresponding to the text sample are not one-to-one, the text sample and the pronunciation phoneme information corresponding to the text sample are fused by an alignment fusion method in an attention mechanism to make the text sample and the pronunciation phoneme information corresponding to the text sample one-to-one. Then the fused information is mapped to word-level focus classification information for focus classification information prediction. The prediction methods include:

[0056] The sample focus classification information corresponding to the predicted continuous representation vector can be predicted in an autoregressive manner, and a mean square loss function MSE loss is used to make the predicted continuous representation vector consistent with the continuous representation vector z extracted by the focus extraction model as a target. c (x) When close, i.e., the focus prediction model reaches the convergence condition, stop training, and then determine the focus category of the closest codebook through the continuous representation vector of the completed training, as the predicted discrete representation vector, i.e., the output result of the focus prediction model according to the text sample.

[0057] The non-autoregressive manner can also be used to make the focus prediction model directly predict the discrete representation vector, and the discrete representation vector is directly classified into the focus category in the codebook through a classification network. Then the prediction cross-entropy loss is used to make the predicted discrete representation vector consistent with the discrete representation vector z extracted by the focus extraction model as a target. d (x) When consistent, i.e., the continuous representation vector z c (x) The closest e i is composed of z d (x) When consistent, i.e., the focus prediction model reaches the convergence condition, stop training. Then the predicted discrete representation vector is taken as the output result of the completed focus prediction model.

[0058] It can be understood that in order to better fuse the text information and the phoneme information, the attention mechanism attention structure is used for alignment of the text information and the phoneme information, and an attention loss function attention smaloss is added. When the attention loss function is smallest, the fusion is stopped, thereby enhancing the monotony of the attention alignment.

[0059] Please refer to Figure 5 , Figure 5is a flowchart of a third embodiment of the voice synthesis method based on focus information of the present application, and the step S100 specifically comprises steps S110 and S120:

[0060] S110, generating initial focus hierarchical information of the text to be synthesized based on the focus prediction model, wherein the focus prediction model is generated according to text samples and sample focus hierarchical labels corresponding to the text samples;

[0061] S120, regulating the initial focus hierarchical information according to the focus regulation instruction to generate the target focus hierarchical information.

[0062] In this embodiment, the text samples used to train the focus prediction model and the sample focus hierarchical labels corresponding to the text samples conform to the language habits of most users. In order to make the target focus hierarchical information more consistent with the language habits of the target user, the initial focus hierarchical information of the text to be synthesized generated by the focus prediction model can be further regulated according to the focus regulation instruction. The focus regulation instruction includes but is not limited to a regulation instruction for manually specifying the position of the focus, or a regulation instruction for regulating the initial focus hierarchical information according to preset prior knowledge. The prior knowledge can be focus hierarchical information extracted by the focus extraction model according to the voice samples of the target user, which conforms to the language habits of the target user. The prior knowledge can also be an intensity scale instruction triggered by the target user's adjustment operation or triggered by the voice synthesis model when detecting that the initial focus hierarchical information meets the regulation condition.

[0063] Therefore, the voice synthesis method based on focus information in this embodiment includes three parts: focus information extraction, focus information prediction, and focus information regulation, thereby realizing voice synthesis with controllable focus. The focus information extraction starts from the local prosodic features of the voice sample, that is, first obtains the local prosodic representation of the voice sample by using an unsupervised network and an adversarial training network, and then extracts the focus hierarchical information at the word level or the character level by using an unsupervised model based on the local prosodic representation. The focus information prediction starts from the text, takes the focus hierarchical information extracted from the voice as the training target of the focus prediction model, and learns the focus hierarchical information in the text sample by using an unsupervised language model. Finally, the focus hierarchical information of the text to be synthesized predicted by the focus prediction model is regulated according to the prior knowledge or the adjustment instruction of the target user, or the intensity scale is performed according to the scaling operation, so that the target focus hierarchical information is more consistent with the language habits of the target user, greatly improving the prosodic naturalness of the synthesized voice, highlighting the related text in the text to be synthesized, and highlighting the key points of the synthesized voice.

[0064] Further, the S120 comprises:

[0065] When the focus control instruction is a focus level setting instruction, the focus position and focus level in the initial focus level information are controlled according to the focus level category of each type of text defined in the focus level setting instruction.

[0066] In this embodiment, the index index of the Codebook of the VQVAE model is manually specified, that is, the focus level information of each word of the predicted text is manually specified. For example, a focus strong representation is set at a position that needs to be emphasized, and a focus weak representation is set at a position that needs to be weakened. In this way, by collocating the focus strong and weak information, the synthesized speech is controlled to emphasize or weaken at the specified position, and the prosody fluctuation of the synthesized speech is enhanced. Specifically, by editing the selection of different code table codebook index, the meaning represented by different discrete representation vectors in the codebook is set. For example, the strong, medium and weak categories in the focus level information. When synthesizing speech, the discrete representation vectors generated by the vqvae model are all selected from the vector categories representing focus weak in the codebook, and then a synthesized speech with low pitch, low prosody fluctuation and fast speech speed is obtained. If the discrete representation vectors generated by the vqvae model are all selected from the vector categories representing focus strong in the codebook, then a synthesized speech with high pitch, high prosody fluctuation and slow speech speed is obtained.

[0067] Further, the S120 comprises:

[0068] When the focus control instruction is a focus intensity scaling instruction, the focus intensity of each type of focus in the initial focus level information is controlled according to the scaling parameters of each type of focus in the focus intensity scaling instruction.

[0069] In this embodiment, when the user needs to adjust the focus intensity, such as further enhancing the text corresponding to the position of the focus strong representation that needs to be emphasized, or weakening the text corresponding to the position of the focus strong representation, the focus intensity scaling instruction can be triggered. When the voice synthesis model detects that the focus intensity difference of the initial focus level information corresponding to the to-be-synthesized text is small, resulting in a small prosody fluctuation of the synthesized speech corresponding to the to-be-synthesized text, the focus intensity scaling instruction can be triggered to further enhance the focus strong representation, and the focus weak representation can also be further weakened. Specifically, each e i is used to represent the focus strong, medium and weak categories. For example, the code table of the VQVAE model is composed of e1, e2 and e3, and e1, e2 and e3 represent the focus information strong, medium and weak categories, respectively. Scaling up e1, e2 and e3 can enhance the focus performance, and the formula is as follows:

[0070]

[0071] Wherein, e1, e2, e3 represent the intensity of the focus point decreases in turn, then by scaling the size of e1, such as scale<1, control the focus intensity of e1 to weaken, that is, e1 approaches e2, compared with before adjustment, the fundamental frequency of e1 is lower, the pronunciation length is longer and shorter; such as scale>1, control the focus intensity of e1 to enhance, compared with before adjustment, the fundamental frequency of e1 is higher, the pronunciation length is longer and grows. Thus, according to the scaling parameters of various types of focus points in the focus intensity scaling instruction, the focus point categories in the code table are scaled, and the prosody fluctuation of the synthesized speech is further enhanced.

[0072] Further, the S120 comprises:

[0073] When the focus point control instruction is a focus point frequency adjustment instruction, the frequencies of various types of focus points in the initial focus point hierarchical information are controlled according to the frequency control parameters of various types of focus points in the focus point frequency adjustment instruction.

[0074] In this embodiment, the focus point hierarchical information corresponding to the target user is extracted from the language sample of the target user according to the focus point extraction model, the strong and weak representations of the focus point hierarchy are analyzed, the proportions of the hierarchical information corresponding to the intensity of various types of focus points are determined, and the number of words corresponding to the division of long pause L3 and short pause L1 in the focus point hierarchical information corresponding to the target user is determined. Then, according to the analysis result, the threshold value in the long and short pause range is adjusted for the predicted cross-entropy softmax value in the initial focus point hierarchical information. Thus, by adjusting the threshold value, the frequency of the appearance of the strong and weak representations of the focus point and the continuous number of words of the strong representation of the focus point are controlled, so that the target focus point hierarchical information is more in line with the language habits of the target user.

[0075] It can be understood that in addition to regulating the initial focus point hierarchical information generated by the focus point prediction model, the trained focus point prediction model can also be regulated, so that the focus point hierarchical information generated by the focus point prediction model has the same distribution as the focus point hierarchical information of the target user. According to the focus point hierarchical information of the target user as prior information, the training of the focus point prediction model is controlled by the focus point.

[0076] First, the strong, medium and weak probability distributions corresponding to each word in the text sample are predicted by the focus point prediction model. When there are N words in the text sample, the prediction result of the focus point prediction model is N probability distributions, that is, p (1) ,p (2) ,...,p (N) The prediction result with a prediction probability greater than a preset threshold r is taken as a high-confidence prediction result, and the prediction result less than or equal to the preset threshold r is taken as a low-confidence prediction result. The softmax distribution probability or the selection entropy is taken as an index for measuring the confidence.

[0077] In order to make the focus prediction model more consistent with the real focus ranking information of the target user, the real prior distribution corresponding to the real focus ranking information of the target user is used to adjust the probability value with low confidence The prediction result p with low confidence (k) is corrected, where n represents the number of words in the sample, The average probability mean is represented as:

[0078]

[0079] Therefore, the probability value with low confidence is adjusted according to the real prior distribution, so as to control the distribution of the predicted focus result according to the focus ranking information of the target user, that is, to control the focus position and the focus frequency in the text.

[0080] Further, the user can increase or decrease the frequency of focus occurrence according to actual needs to control the focus fluctuation of the synthesized speech. When the prosody fluctuation does not meet the condition, the speech synthesis model can increase or decrease the frequency of focus occurrence, so as to control the focus fluctuation of the synthesized speech.

[0081] Specifically, the threshold value adjustment of the predicted cross-entropy softmax value in the long and short pause range can be performed by dividing the long pause L3 and the short pause L1 before speech synthesis and the number of related words. For example, no focus strong representation is allowed to appear in the L3 range of less than N words, or multiple focus strong representations are allowed to appear in the L3 range of more than M words. In this way, the focus frequency in the synthesized speech is increased or decreased.

[0082] In specific embodiments, the focus frequency control can also be performed according to the real prior distribution of the target user. For example, the prior distribution of the target user is converted to a preset prior distribution For the prediction result with low confidence, the distribution of the preset prior distribution is used for adjustment, so as to increase or decrease so as to adjust the probability value with low confidence to be close to the preset prior.

[0083] Therefore, by the three parts of focus information extraction, focus information prediction, and focus information control, a focus-controllable speech synthesis model is constructed. The focus ranking information is extracted from the local prosodic representation of the speech sample, and the focus ranking information is added to the extraction model of the local prosodic representation, so as to better recover the target local prosodic representation with focus ranking information according to the focus ranking information, and then control the speech synthesis through the target local prosodic representation. Through the focus prediction model and the focus regulation, the target focus ranking information more consistent with the target user is obtained.

[0084] In addition, as a specific implementation of the method shown in Figure 2 The embodiment of the present application provides a speech synthesis device based on focus information.​

[0085] Referring to Figure 6 , Figure 6 is a functional module schematic diagram of a voice synthesis device based on focus information provided by an embodiment of the present application. The device comprises:

[0086] a focus information prediction module 10, configured to generate target focus hierarchical information of a text to be synthesized based on a focus prediction model, wherein the focus prediction model is generated according to a text sample and a sample focus hierarchical label corresponding to the text sample;

[0087] a prosody representation generation module 20, configured to obtain a target prosody label of the text to be synthesized, and generate a target local prosody representation of the text to be synthesized according to the text to be synthesized, the target focus hierarchical information and the target prosody label;

[0088] a target voice synthesis module 30, configured to generate a target synthesized voice of the text to be synthesized according to the target local prosody representation.

[0089] Further, the focus information prediction module 10 comprises:

[0090] an initial focus hierarchical unit, configured to generate initial focus hierarchical information of the text to be synthesized based on the focus prediction model, wherein the focus prediction model is generated according to a text sample and a sample focus hierarchical label corresponding to the text sample;

[0091] an initial focus regulation unit, configured to regulate the initial focus hierarchical information according to a focus regulation instruction to generate the target focus hierarchical information.

[0092] Further, the initial focus regulation unit further comprises:

[0093] a focus hierarchical setting subunit, configured to, when the focus regulation instruction is a focus hierarchical setting instruction, regulate a focus position and a focus category in the initial focus hierarchical information according to a focus category of each type of text defined in the focus hierarchical setting instruction.

[0094] Further, the initial focus regulation unit further comprises:

[0095] a focus intensity regulation subunit, configured to, when the focus regulation instruction is a focus intensity scaling instruction, regulate each focus intensity in the initial focus hierarchical information according to a scaling parameter of each focus in the focus intensity scaling instruction.

[0096] Further, the initial focus regulation unit further comprises:

[0097] The focus frequency regulation subunit is configured to, when the focus regulation instruction is a focus frequency adjustment instruction, regulate the frequency of each type of focus in the initial focus hierarchical information according to the frequency control parameters of each type of focus in the focus frequency adjustment instruction.

[0098] Further, the device further comprises a prediction model training module, configured to:

[0099] obtain a speech sample corresponding to the text sample, and generate a sample prosody representation of the speech sample according to a prosody representation generation model;

[0100] extract sample focus hierarchical information from the sample prosody representation as the sample focus hierarchical label;

[0101] train the focus prediction model according to the text sample and the sample focus hierarchical label.

[0102] Further, the prediction model training module is further configured to:

[0103] encode the sample prosody representation into a continuous representation vector through a focus extraction model;

[0104] quantize the continuous representation vector into a discrete representation vector for representing a focus category according to the distance between each focus category vector in a preset code table and the continuous representation vector, as the sample focus hierarchical information.

[0105] In addition, the embodiment of the present application further provides a computer readable storage medium.

[0106] The computer readable storage medium of the present application stores a focus information based speech synthesis program, wherein the focus information based speech synthesis program is executed by a processor to realize the steps of the focus information based speech synthesis method as described above. The method realized by the execution of the focus information based speech synthesis program can refer to each embodiment of the focus information based speech synthesis method of the present application, and will not be described here.

[0107] It should be understood that the terms used in this application are only for the purpose of describing specific embodiments and are not intended to limit the present application.

[0108] It should also be understood that the term "and / or" used in the present application and the appended claims means any combination of one or more of the associated listed terms and all possible combinations, and includes these combinations.

[0109] It should be noted that the terms "first", "second", etc. in the embodiments of the present application are used only for the purpose of description, and should not be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined as "first", "second" can explicitly or implicitly include at least one of the features.

[0110] The above description is merely a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or replacements within the technical scope disclosed by the present application, and these modifications or replacements should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speech synthesis method based on focus information, characterized in that: The method comprises: Based on the focus prediction model, initial focus classification information of the text to be synthesized is generated, wherein the training process of the focus prediction model includes: Decoupling the speech samples corresponding to the text samples and removing the decoupled text information and speaker information to obtain a local prosodic representation that represents word-level prosodic changes; quantizing the local prosody representation into a discrete representation vector as a sample focus classification label; Training the focus prediction model based on the text samples and the sample focus classification labels; Dynamically regulating the initial focus grading information according to a focus regulation instruction to obtain target focus grading information; the focus regulation instruction includes a focus intensity scaling instruction and / or a focus frequency adjustment instruction, and the focus regulation instruction is generated based on the language habits of the target user; obtaining a target prosodic label of the text to be synthesized, and generating a target local prosodic representation of the text to be synthesized based on the text to be synthesized, the target focus grading information, and the target prosodic label; A target synthesized speech of the text to be synthesized is generated according to the target local prosody representation.

2. The speech synthesis method according to claim 1, wherein: The dynamically regulating the initial focus classification information according to the focus regulation instruction includes: When the focus control instruction is a focus grading setting instruction, the focus position and focus category in the initial focus grading information are controlled according to the focus categories to which various types of texts belong as defined in the focus grading setting instruction.

3. The speech synthesis method according to claim 2, wherein: The dynamically regulating the initial focus classification information according to the focus regulation instruction includes: When the focus control instruction is a focus intensity scaling instruction, the various focus intensities in the initial focus grading information are regulated according to the scaling parameters of the various focus types in the focus intensity scaling instruction.

4. The speech synthesis method according to claim 2, wherein: The dynamically regulating the initial focus classification information according to the focus regulation instruction includes: When the focus control instruction is a focus frequency adjustment instruction, the frequencies of various types of focuses in the initial focus grading information are adjusted according to the frequency control parameters of the various types of focuses in the focus frequency adjustment instruction.

5. The speech synthesis method according to any one of claims 1 to 4, characterized in that: Before generating the initial focus grading information of the text to be synthesized based on the focus prediction model, the method further includes: Obtaining a speech sample corresponding to the text sample, and generating a sample prosodic representation of the speech sample based on a prosodic representation generation model; Extracting sample focus grading information from the sample prosody representation as the sample focus grading label; The focus prediction model is trained and generated based on the text samples and the sample focus grading labels.

6. The speech synthesis method according to claim 5, characterized in that: The extracting sample focus classification information from the sample prosody representation includes: encoding the sample prosody representation into a continuous representation vector through a focus extraction model; According to the distance between each focus category vector in the preset code table and the continuous characterization vector, the continuous characterization vector is quantized into a corresponding discrete characterization vector for representing the focus category as the sample focus grading information.

7. A speech synthesis device based on focus information, characterized in that: The device comprises: A focus information prediction module is used to generate initial focus classification information of the text to be synthesized based on a focus prediction model, wherein the training process of the focus prediction model includes: Decoupling the speech samples corresponding to the text samples and removing the decoupled text information and speaker information to obtain a local prosodic representation that represents word-level prosodic changes; quantizing the local prosody representation into a discrete representation vector as a sample focus classification label; Training the focus prediction model based on the text samples and the sample focus classification labels; Dynamically regulating the initial focus grading information according to a focus regulation instruction to obtain target focus grading information; the focus regulation instruction includes a focus intensity scaling instruction and / or a focus frequency adjustment instruction, and the focus regulation instruction is generated based on the language habits of the target user; a prosodic representation generating module, configured to obtain a target prosodic label of the text to be synthesized, and generate a target local prosodic representation of the text to be synthesized based on the text to be synthesized, the target focus classification information, and the target prosodic label; The target speech synthesis module is used to generate the target synthesized speech of the text to be synthesized according to the target local prosody representation.

8. A speech synthesis device based on focus information, characterized in that: The speech synthesis device includes a processor, a memory, and a speech synthesis program stored in the memory and executable by the processor, wherein when the speech synthesis program is executed by the processor, the steps of the speech synthesis method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, wherein when the computer program is executed by a processor, the steps of the speech synthesis method according to any one of claims 1 to 6 are implemented.

Images