Speech synthesis method and device, electronic device and storage medium

Abstract

The present application discloses a speech synthesis method and device, an electronic device and a storage medium, wherein the speech synthesis method comprises: extracting pronunciation attribute features of a text to be synthesized, and obtaining target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; and then synthesizing the synthesized speech of the text to be synthesized based on the pronunciation attribute features and the target attribute features. The above scheme can reduce the cost while improving the freedom of speech synthesis.

Description

Technical Field

[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method and device, electronic equipment and storage medium. Background Art

[0002] Speech synthesis technology is one of the core technologies for realizing human-computer interaction. With the continuous development and improvement of speech synthesis technology, speech synthesis has been widely used in all aspects of social life, including public services (information broadcasting, intelligent customer service, etc.), smart hardware (smart speakers, smart robots, etc.), smart transportation (voice navigation, smart car equipment, etc.), education (smart classroom, foreign language learning, etc.), pan-entertainment (audio reading, film and television dubbing, virtual images, etc.) and other fields.

[0003] At present, speech synthesis technology has been able to reach a level close to that of real people, and improving the freedom of synthesized speech and reducing the cost of building a synthesis system have become one of the current research hotspots in speech synthesis technology. Although this problem can be solved by having the same speaker record speech data with different emotions and styles for modeling, this method requires a high level of speaker ability, and it is difficult to find speakers and record audio. In view of this, how to improve the freedom of speech synthesis while reducing its cost has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a speech synthesis method and device, an electronic device and a storage medium, which can improve the freedom of speech synthesis while reducing its cost.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a speech synthesis method, including: extracting pronunciation attribute features of a text to be synthesized; and based on the target categories of the text to be synthesized in several speech attributes, obtaining target attribute features of various speech attributes; and then synthesizing the synthesized speech of the text to be synthesized based on the pronunciation attribute features and the target attribute features.

[0006] In order to solve the above technical problems, the second aspect of the present application provides a speech synthesis method, including: extracting pronunciation attribute features of the speech to be processed; obtaining target attribute features of various speech attributes based on the target categories of the speech to be processed in several speech attributes; and then synthesizing the synthesized speech of the speech to be processed based on the pronunciation attribute features and the target attribute features.

[0007] In order to solve the above-mentioned technical problems, the third aspect of the present application provides a speech synthesis device, including an extraction module, an acquisition module and a synthesis module. The extraction module is used to extract pronunciation attribute features of a text to be synthesized; the acquisition module is used to acquire target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; the synthesis module is used to synthesize the synthesized speech of the text to be synthesized based on the pronunciation attribute features and the target attribute features.

[0008] In order to solve the above-mentioned technical problems, the fourth aspect of the present application provides a speech synthesis device, including an extraction module, an acquisition module and a synthesis module, the extraction module is used to extract the pronunciation attribute features of the text to be synthesized; the acquisition module is used to acquire the target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; the synthesis module is used to synthesize the synthesized speech of the text to be synthesized based on the pronunciation attribute features and the target attribute features.

[0009] In order to solve the above technical problems, the fifth aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the speech synthesis method in the above first aspect or second aspect.

[0010] In order to solve the above technical problems, the sixth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, wherein the program instructions are used to implement the speech synthesis method in the above first aspect or second aspect.

[0011] The above scheme extracts the pronunciation attribute features of the text to be synthesized, and then obtains the target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; finally, the synthesized speech of the text to be synthesized is synthesized based on the pronunciation attribute features and the target attribute features. On the one hand, since there is no need for manual large-scale speech recording, time is saved, which helps to reduce the cost of speech synthesis. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized in several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis, so that speech data with any category of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application;

[0013] Figure 2 yes Figure 1 Schematic diagram of attribute feature extraction in step S12 of an embodiment;

[0014] Figure 3 yes Figure 1 Schematic diagram of attribute feature synthesis in step S13 of an embodiment;

[0015] Figure 4 is a flowchart of another embodiment of the speech synthesis method of the present application;

[0016] Figure 5 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application;

[0017] Figure 6 is a schematic diagram of the framework of another embodiment of the speech synthesis device of the present application;

[0018] Figure 7 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application;

[0019] Figure 8 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0023] See also Figure 1 , Figure 1 It is a flowchart of an embodiment of the speech synthesis method of the present application.

[0024] Specifically, the following steps may be included:

[0025] Step S11: extracting pronunciation attribute features of the text to be synthesized.

[0026] In an implementation scenario, the pronunciation attributes may include but are not limited to pronunciation content information, etc., which is not limited here. Specifically, the pronunciation sequence of the text to be synthesized can be extracted first, and the pronunciation sequence contains several pronunciation markers, and then each pronunciation marker is feature-encoded respectively, so as to obtain the pronunciation attribute characteristics of the text to be synthesized. Exemplarily, the International Phonetic Alphabet (IPA) can be used to mark the pronunciation content information of the text to be synthesized to obtain a pronunciation sequence; in addition, one-hot encoding can be used to feature encode each pronunciation marker. For the specific process, please refer to technical details such as IPA and one-hot, which will not be repeated here.

[0027] Step S12: based on the target categories of the text to be synthesized in several speech attributes, target attribute features of various speech attributes are obtained.

[0028] In one implementation scenario, the plurality of speech attributes include at least one of channel attributes, dialect attributes, timbre attributes, emotion attributes, and style attributes. The above method, by distinguishing speech attributes, that is, distinguishing coupled speech information into multiple speech attributes by decoupling, helps to improve the diversity of speech synthesis in the speech synthesis process and improves the efficiency of speech synthesis.

[0029] In an implementation scenario, the difficulty in controlling speech attributes lies in the fact that speech contains a lot of information coupling. In order to realize the free combination of attribute information contained in speech and control the synthesis, it is necessary to distinguish the speech attributes and decouple the coupled information, so as to realize the free combination and control of speech synthesis content. Speech includes rich attribute information. In order to realize the control of implementation information, it is first necessary to determine the category of speech attributes. The specific category of speech attributes can be set according to the specific application scenario, and no specific restrictions are made here. This application takes the channel attributes, dialect attributes, timbre attributes, emotional attributes, and style attributes in speech attributes as examples to illustrate the speech synthesis process.

[0030] In a specific implementation scenario, the channel attribute refers to the environment in which the voice is located, such as a recording studio, a conference room, or a car. In different recording environments, the channel attributes of the voice are different, and the listening experience is also different. By controlling the voice channel attributes to generate voice, the synthesized voice can be more consistent with the scene and the synthesized voice can be more realistic; the dialect attribute refers to different dialects under a language, such as Mandarin, Northeastern dialect, etc. under Chinese, or British English, American English, etc. under English. The dialect category is determined according to the language scenario used; the timbre attribute refers to the voice timbre information contained in the voice to distinguish the identity of the speaker; the emotional attribute refers to the emotional category contained in the voice, such as neutral, happy, sad, angry, etc.; the style attribute refers to the speech style of the voice, such as the speech style in different scenarios such as news, interaction, customer service, and novels. After the voice attributes are determined, several voice attributes can also be subdivided into categories. For example, for the timbre attribute, each speaker can be used as a category, gender can be used as a category, and age range can be used as a category. The specific classification of the timbre attribute can be set according to the actual situation, and no specific limitation is made here. For attributes with mature classifications such as emotions, they are determined according to application requirements. For attributes with mature classifications such as emotions, they can be determined based on application requirements.

[0031] In an implementation scenario, speech attributes include multiple categories. To ensure the accuracy of speech synthesis, speech data needs to be collected. Speech data is collected based on multiple categories of speech attributes to cover these speech attribute categories. To ensure the accuracy of speech attributes between different categories when decoupling, it should be ensured that speech data of at least two people are collected for each attribute category in several speech attributes. In this process, the speech data can be recorded by the speaker; the speech data can also be obtained through collection and other means; the speech data can also be obtained by processing the recording data in the recording equipment of the announcement place. The specific method of collecting speech data can be selected according to the actual application scenario, and no specific restrictions are made here.

[0032] In one implementation scenario, according to the target categories of the text to be synthesized on several speech attributes, the target attribute features of various speech attributes are obtained, which at least includes: feature modeling based on the speech data related to the attribute categories of various speech attributes to obtain the attribute space probability distribution model of various speech attributes; and then sampling in the attribute space probability distribution model of the corresponding speech attribute based on the target category of the text to be synthesized on the speech attribute to obtain the target attribute features of the corresponding speech attribute. In the above method, the attribute space probability distribution model of the speech attribute can be obtained by feature modeling the speech data related to the attribute category of the speech attribute. Different attribute categories can be divided into regions in the spatial probability distribution model to obtain the distribution space of different attribute categories under the same speech attribute, which is helpful for sampling data in the speech synthesis process and improves the efficiency of speech synthesis.

[0033] In one implementation scenario, each speech attribute has at least one attribute category; then feature modeling is performed based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes, including: for various attribute categories under speech attributes, sample attribute features of speech data related to the attribute category are extracted; then based on the sample attribute features of various attribute categories under speech attributes, the attribute space probability distribution model of speech attributes is constructed. In the above method, by constructing the attribute space probability distribution model of speech attributes, the corresponding attribute features can be directly sampled in the distribution model during the speech synthesis process, thereby improving the efficiency of speech synthesis.

[0034] In an implementation scenario, in the process of obtaining target attribute features of various speech attributes, the target attributes can be extracted by manual annotation, or feature modeling can be performed on speech data related to each speech attribute category to obtain an attribute space probability distribution model of various speech attributes, and the target attribute features can be obtained by sampling the attribute space probability distribution of the target speech attribute. The specific method for obtaining the target attribute features can be selected according to the actual application scenario and is not specifically limited here.

[0035] In a specific implementation scenario, before performing feature modeling on speech data related to attribute categories of various speech attributes, it is necessary to extract the features of each attribute category in the speech data, and then model the attribute features to obtain the attribute space probability distribution model of various speech attributes. In the process of extracting the target attribute, the target attribute can be extracted by establishing a feature extraction model, or by manual annotation. The specific method of extracting the target attribute can be set according to the actual situation and is not specifically limited here.

[0036] In a specific implementation scenario, the target attribute category features are extracted by manual annotation. The manual annotation method refers to that a person annotates the corresponding attribute features after listening to the speech, and then encodes the annotated information by a certain encoding method, and uses the encoded features as the representation features of the attribute information. For example: if the speech attribute includes the pronunciation attribute, the pronunciation attribute can be annotated with the International Phonetic Alphabet (IPA) to obtain the pronunciation sequence annotation of the speech, and then each pronunciation in the pronunciation sequence is encoded by using a unique hot encoding to obtain the encoding representation sequence. Alternatively, the emotional attribute is manually annotated, and the emotional attribute includes neutral, happy, sad, angry, etc. The emotional attribute can be annotated by a unique hot encoding, neutral can be annotated as 1000, happy can be annotated as 0100, sad can be annotated as 0010, and angry can be annotated as 0001; the emotional attribute can also be encoded by Arabic numerals, neutral can be annotated as 1, happy can be annotated as 2, sad can be annotated as 3, and angry can be annotated as 4. In the process of manually annotating the target attribute, different encoding methods can be selected for annotation according to the actual situation, and no specific limitation is made here.

[0037] In a specific implementation scenario, the target attribute category features are extracted by establishing features and then extracting the model by a model training method. The model training method may include but is not limited to supervised learning, self-supervised learning, modeling methods, etc. Among them, in the learning method of the supervised learning method, the existing attribute annotation data is first used as training data to train the sentiment classification model. For example: taking the sentiment attribute as an example, the training data of the sentiment attribute can be recorded voice data or additionally collected voice data with sentiment annotations. The sentiment classification model can be used to predict the sentiment of the voice data to obtain the predicted sentiment category of the voice data, and based on the difference between the sample sentiment category annotated with the voice data and the predicted sentiment category, the network parameters of the sentiment classification model are adjusted to realize the training of the sentiment classification model. It should be noted that the sentiment classification model can adopt a deep neural network, a convolutional neural network, etc., which is not limited here. In addition, the model can be continuously updated and optimized by minimizing the cross entropy loss function to obtain a sentiment classification model with convergence in training. In this case, the last layer of hidden features of the sentiment classification model can be used as the sentiment attribute features of the current speech. In addition, in order to ensure that the extracted emotional attribute features do not contain feature information of other attributes, additional information constraint criteria can be designed during the training of the emotional classification model, such as minimizing the mutual information with timbre, to constrain the extracted information representation to be unaffected by other attribute features. The same can be said for other speech attributes, which will not be listed one by one here.

[0038] See also Figure 2 , Figure 2 yes Figure 1 Schematic diagram of attribute feature extraction in step S12 of an embodiment of FIG. Figure 2 As shown, the voice data 20 is audio information collected in a variety of ways, and several attribute modules may include a channel attribute module 21, a dialect attribute module 22, a timbre attribute module 23, an emotion attribute module 24 and a style attribute module 25, and each attribute module may be obtained based on a method of manual annotation or model learning. Of course, several attribute modules may also include a pronunciation attribute module (not shown), through which pronunciation attribute features may be extracted. The specific extraction process of the pronunciation attribute module may refer to the aforementioned description of extracting pronunciation attribute features, which will not be repeated here. Corresponding to each attribute module, the attribute features may include a channel attribute feature 210, a dialect attribute feature 220, a timbre attribute feature 230, an emotion attribute feature 240 and a style attribute feature 250, and the attribute features are obtained according to the corresponding attribute module. After a large amount of attribute feature extraction corresponding to voice data, different attribute category features corresponding to different attribute categories in each attribute may be obtained. According to the sample attribute features of various attribute categories under voice attributes, an attribute space probability distribution model of voice attributes is constructed. According to the attribute space probability distribution model of voice attributes, corresponding attribute features of different attribute categories may be obtained.

[0039] In an implementation scenario, feature modeling is performed based on speech data related to attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes. Methods for modeling attribute space probability distribution models of speech attributes may include but are not limited to modeling methods of maximum likelihood criteria, etc., wherein the model structure adopted includes but is not limited to Gaussian mixture models, flow models, etc. After the attribute space probability distribution model of speech attributes is established, during the speech synthesis process, the probability distribution can be used to sample different attribute categories to obtain target attribute category features. In addition, in order to achieve controlled sampling of attribute space probability distribution, other fine control information is input during modeling to model the conditional probability distribution of attribute feature space. For example, for the emotional space, when modeling, the emotional category information is input into the model for modeling. Thus, in the generation stage, the specified emotional attribute category features can be sampled, so that more refined sampling predictions can be achieved.

[0040] In a specific implementation scenario, the attribute features of the attribute space probability distribution model of speech attributes need to be obtained from the speech through the model learning method, and when establishing a model for the attribute space probability distribution of speech attributes, the speech attributes can be modeled selectively, and the specific selection method can be selected according to the specific situation.

[0041] Step S13: Based on the pronunciation attribute features and each target attribute feature, synthesize the synthesized speech of the text to be synthesized.

[0042] In an implementation scenario, the pronunciation attribute features are the pronunciation attribute features of the text to be synthesized, and each target attribute feature is a target attribute feature sampled in an attribute space probability distribution model of various speech attributes. All the acquired attribute features are synthesized to obtain synthesized speech.

[0043] In an implementation scenario, as mentioned above, the speech attributes include timbre attributes. Then, before synthesizing the synthesized speech of the text to be synthesized based on the pronunciation attribute features and each target attribute feature, it is also possible to obtain reference attribute features of the timbre attributes based on the target category of the text to be synthesized in terms of the timbre attributes; and the reference attribute features of the timbre attributes are extracted based on speech data related to the target category of the timbre attributes; and then, adjustment processing is performed based on the reference attribute features of the timbre attributes to obtain target attribute features of the timbre attributes, so that the timbre of the speaker in the training set can be generated, and on this basis, the synthesized speech of the text to be synthesized can be synthesized based on the pronunciation attribute features and each target attribute feature. In the above manner, during the speech synthesis process, the reference attributes can also be adjusted to generate the timbre of the speaker in the training set, which is conducive to quickly creating a synthesized speech containing rich timbre.

[0044] In a specific implementation scenario, during the adjustment process based on the reference attribute features of the timbre attribute, multiple reference attribute features of the timbre attribute may be weighted to obtain the target attribute features of the timbre attribute. For example, by sampling the timbre space probability distribution, a synthetic speech different from the timbre of the speaker in the training set can be generated, thereby generating a synthetic speech with a new timbre. A and S B , a new timbre attribute feature S can be generated by interpolation new , by adding weight λ, where λ is the interpolation weight between two features, and the value range is 0<λ<1. new New speech data can be synthesized by speech, which can be expressed as S new =λ*S A +(1-λ)*S B The above method, using this method to construct synthetic speech, saves the work of finding a speaker and recording audio, can generate synthetic speech with a timbre different from that of the speaker in the training set, thereby generating synthetic speech with a new timbre, and has the advantage of avoiding copyright risks.

[0045] In a specific implementation scenario, during the adjustment process based on the reference attribute feature of the timbre attribute, the reference attribute feature of the timbre attribute may also be scaled to obtain the target attribute feature of the timbre attribute. C , a new timbre attribute feature S can be generated by stretching new , by adding weight λ, the stretched feature S new It can be expressed as S new =λ*S C The above method, by constructing a synthetic speech in this way, can generate a synthetic speech with a timbre different from that of the speaker in the training set, thereby generating a synthetic speech with a new timbre, which can reduce the cost of speech synthesis.

[0046] See also Figure 3 , Figure 3 yes Figure 1 The schematic diagram of attribute feature synthesis in step S13 of an embodiment is as follows: Figure 3 As shown, the speech attribute features include pronunciation attribute features 30, channel attribute features 31, dialect attribute features 32, timbre attribute features 33, emotion attribute features 34 and style attribute features 35. The pronunciation attribute features and the above-mentioned speech attribute features can be input into the speech synthesis module 36 for synthesis processing to obtain synthesized speech. Specifically, the speech synthesis module 36 can be trained by sample speech. In this training process, the pronunciation attribute features of the sample speech and the above-mentioned speech attribute features can be obtained, and the pronunciation attribute features of the sample speech and the above-mentioned speech attribute features can be synthesized by the speech synthesis module 36 to obtain the synthesized speech corresponding to the sample speech, so that the network parameters of the speech synthesis module 36 can be adjusted based on the difference between the sample speech and its corresponding synthesized speech (such as the difference between the Mel spectra of the two), so that the speech synthesis module 36 can be trained to realize the control generation of speech by each attribute feature. It should be noted that the input of the speech synthesis module 36 can be the attribute features extracted above, and the output is the synthesized speech 37. The hidden layer network is composed of a deep neural network module, for example, it can be a combination of one or more of the network types such as a deep neural network, a recurrent neural network, and a convolutional neural network. Then the network is trained by setting certain training criteria, including but not limited to minimum mean square error criteria, maximum likelihood criteria, etc. According to the speech synthesis module 36 , speech synthesis can be controlled, and the characteristics of the synthesized speech 37 can be better controlled.

[0047] The above scheme extracts the pronunciation attribute features of the text to be synthesized, and then obtains the target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; finally, the synthesized speech of the text to be synthesized is synthesized based on the pronunciation attribute features and the target attribute features. On the one hand, since there is no need for manual large-scale speech recording, time is saved, which helps to reduce the cost of speech synthesis. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized in several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis, so that speech data with any category of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0048] See also Figure 4 , Figure 4 It is a flowchart of another embodiment of the speech synthesis method of the present application. It should be noted that the embodiment of the present disclosure can perform arbitrary attribute migration on the basis of the speech to be processed, so as to synthesize a synthesized speech having some of the same attribute characteristics as the speech to be processed. In addition, the embodiment of the present disclosure only focuses on the differences from the aforementioned disclosed embodiment. The same or similar aspects can be referred to the aforementioned disclosed embodiment, which will not be repeated here. Specifically, the embodiment of the present disclosure may specifically include the following steps:

[0049] Step S41: extracting pronunciation attribute features of the speech to be processed.

[0050] In one implementation scenario, as described above, the pronunciation attribute may include pronunciation content information in the speech. Similar to the aforementioned extraction of pronunciation attribute features of the text to be synthesized, for the speech to be processed, the pronunciation content in the speech may be annotated using the International Phonetic Alphabet (IPA) to obtain the pronunciation sequence annotation of the speech, and then each pronunciation in the pronunciation sequence may be encoded using one-hot encoding to obtain the pronunciation attribute features.

[0051] Step S42: based on the target categories of the speech to be processed in several speech attributes, target attribute features of various speech attributes are obtained.

[0052] In one implementation scenario, target attribute features of various speech attributes are obtained based on the target categories of the speech to be processed at several speech attributes. Please refer to the aforementioned disclosed embodiment for the relevant description on obtaining target attribute features of various speech attributes based on the target categories of the text to be synthesized at several speech attributes, which will not be repeated here.

[0053] In one implementation scenario, before obtaining target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes, at least one speech attribute can be selected as the first speech attribute, and the unselected speech attributes can be used as the second speech attribute; based on the target categories of the speech to be processed on several speech attributes, the target attribute features of various speech attributes can be obtained, and based on the target categories of the speech to be processed on each first speech attribute, the target attribute features of each first speech attribute can be obtained; and the speech attribute features of the speech to be processed on each second speech attribute can be extracted as the target attribute features corresponding to the second speech attribute. In the above manner, by migrating the speech attribute features, the attribute feature migration of the speech can be completed, and multiple synthesized speech with different attribute feature categories can be controlled.

[0054] In a specific implementation scenario, taking the selection of emotional attributes as the first voice attribute as an example, the channel attribute, dialect attribute, timbre attribute, and style attribute can be used as the second voice attribute. Each attribute in the to-be-processed voice has its own attribute category, and the target attribute feature of the first voice attribute is happy. At this time, the attribute feature of the emotional attribute with the attribute category of happy can be extracted from the existing voice data, and the attribute feature of the extracted emotional attribute with happy can be synthesized with the attribute features of the remaining attributes to obtain the final synthesized voice. In other words, for the to-be-processed voice, only the emotional attribute feature is changed, and the remaining attribute features are maintained to obtain the final synthesized voice.

[0055] In a specific implementation scenario, taking the selection of style attributes as the first speech attributes as an example, channel attributes, dialect attributes, timbre attributes, and emotional attributes can be used as the second speech attributes. Each attribute in the speech to be processed has its own attribute category, and the target attribute feature of the first speech attribute is interaction. At this time, the attribute feature of the style attribute with the attribute category of interaction can be extracted from the existing speech data, and the extracted attribute feature of the style attribute with the attribute category of interaction can be synthesized with the attribute features of the remaining attributes to obtain the final synthesized speech.

[0056] It should be noted that the above only uses emotional attributes and style attributes as examples to illustrate the specific process of attribute feature migration. In actual application, migration can be performed by selecting any one of the aforementioned speech attributes or a combination of at least two of them, and this is not limited here.

[0057] In addition, exemplarily, the pronunciation attribute features of the speech to be processed can be extracted, and based on the target categories of the speech to be processed in channel attributes, timbre attributes, dialect attributes, emotional attributes, and style attributes, the target attribute features of the above-mentioned speech attributes can be obtained, and the target category of the speech to be processed in timbre attributes is its original timbre category. In other words, the timbre attribute features can be extracted from the speech to be processed as the target attribute features of the timbre attributes, so that speech synthesis can be performed based on the pronunciation attribute features, timbre attribute features, and the target attribute features corresponding to the target categories specified in each speech attribute such as channel attributes, dialect attributes, emotional attributes, style attributes, etc., so as to freely synthesize synthetic speech with different channels, different dialects, different emotions, and different styles while retaining the original pronunciation attributes and original timbre attributes of the speech to be processed.

[0058] Step S43: synthesizing the synthesized speech of the speech to be processed based on the pronunciation attribute features and each target attribute feature.

[0059] In an implementation scenario, a synthesized speech of the speech to be processed can be synthesized based on the pronunciation attribute features and each target attribute feature, and the target attribute feature can be set according to the actual application scenario, which can generate a variety of synthesized speech on the one hand, and effectively control the speech attribute features of the synthesized speech on the other hand. For the specific synthesis process, please refer to the relevant description of synthesizing the synthesized speech of the text to be synthesized based on the pronunciation attribute features and each target attribute feature in the aforementioned disclosed embodiment, which will not be repeated here.

[0060] The above scheme extracts the pronunciation attribute features of the speech to be processed, and then obtains the target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes; finally, the synthesized speech of the speech to be processed is synthesized based on the pronunciation attribute features and the target attribute features. On the one hand, since there is no need for manual large-scale speech recording, time is saved, which helps to reduce the cost of speech synthesis. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized on several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis, so that speech data with any category of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0061] See also Figure 5 , Figure 5It is a schematic diagram of the framework of an embodiment of the speech synthesis device of the present application. The speech synthesis device 50 includes an extraction module 51, an acquisition module 52 and a synthesis module 53. Among them, the extraction module 51 is used to extract the pronunciation attribute features of the text to be synthesized; the acquisition module 52 is used to obtain the target attribute features of various speech attributes based on the target categories of the text to be synthesized on several speech attributes; the synthesis module 53 is used to synthesize the synthesized speech of the text to be synthesized based on the pronunciation attribute features and the target attribute features.

[0062] The above scheme extracts the pronunciation attribute features of the text to be synthesized, and then obtains the target attribute features of various speech attributes based on the target categories of the text to be synthesized in several speech attributes; finally, the synthesized speech of the text to be synthesized is synthesized based on the pronunciation attribute features and the target attribute features. On the one hand, since there is no need for manual large-scale speech recording, time is saved, which helps to reduce the cost of speech synthesis. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized in several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis, so that speech data with any category of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0063] In some disclosed embodiments, the several speech attributes include at least one of channel attributes, dialect attributes, timbre attributes, emotion attributes, and style attributes.

[0064] Therefore, by distinguishing speech attributes, that is, distinguishing coupled speech information into multiple speech attributes by decoupling, it is helpful to improve the diversity of speech synthesis in the speech synthesis process and improve the efficiency of speech synthesis.

[0065] In some disclosed embodiments, the acquisition module 52 includes a feature modeling submodule, which is used to perform feature modeling based on speech data related to attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes; the acquisition module 52 includes a model sampling submodule, which is used to perform sampling in the attribute space probability distribution model of the corresponding speech attribute based on the target category of the speech attribute of the text to be synthesized to obtain the target attribute feature of the corresponding speech attribute.

[0066] Therefore, by performing feature modeling on speech data related to the attribute categories of speech attributes, an attribute space probability distribution model of speech attributes can be obtained. Different attribute categories can be divided into regions in the spatial probability distribution model to obtain the distribution space of different attribute categories under the same speech attributes, which helps in sampling data during speech synthesis and improves the efficiency of speech synthesis.

[0067] In some disclosed embodiments, each speech attribute has at least one attribute category; the feature modeling submodule includes an extraction unit for extracting sample attribute features of speech data related to the attribute category for various attribute categories under the speech attributes; the feature modeling submodule includes a construction unit for constructing an attribute space probability distribution model of the speech attributes based on the sample attribute features of the various attribute categories under the speech attributes.

[0068] Therefore, by constructing an attribute space probability distribution model of speech attributes, corresponding attribute features can be directly sampled in the distribution model during the speech synthesis process, thereby improving the efficiency of speech synthesis.

[0069] In some disclosed embodiments, the synthesis module 53 includes an acquisition submodule for acquiring reference attribute features of the timbre attribute based on the target category of the timbre attribute of the text to be synthesized; and the reference attribute features of the timbre attribute are extracted based on speech data related to the target category of the timbre attribute; the synthesis module 53 includes an adjustment submodule for performing adjustment processing based on the reference attribute features of the timbre attribute to obtain target attribute features of the timbre attribute.

[0070] Therefore, in the process of speech synthesis, the reference attributes can be adjusted to obtain the target attribute characteristics, which can generate synthesized speech with a different timbre from the speaker in the training set, thereby generating synthesized speech with a new timbre and having the advantage of avoiding copyright risks.

[0071] In some disclosed embodiments, the adjustment submodule includes either a weighted processing unit or a scale adjustment unit: the weighted processing unit is used to perform weighted processing on multiple reference attribute features of the timbre attribute to obtain a target attribute feature of the timbre attribute; the scale adjustment unit is used to perform scale adjustment on the reference attribute features of the timbre attribute to obtain a target attribute feature of the timbre attribute.

[0072] Therefore, the timbre attributes can be adjusted through interpolation, stretching, etc., and the target attribute characteristics of the timbre attributes can be obtained, and then in the speech synthesis process, speech data of a new timbre different from the timbre of the speaker in the training set can be obtained, which has the advantage of avoiding copyright risks.

[0073] See also Figure 6 , Figure 6It is a schematic diagram of the framework of another embodiment of the speech synthesis device of the present application. The speech synthesis device 60 includes an extraction module 61, an acquisition module 62 and a synthesis module 63. Among them, the extraction module 61 is used to extract the pronunciation attribute features of the speech to be processed; the acquisition module 62 is used to obtain the target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes; the synthesis module 63 is used to synthesize the synthesized speech of the speech to be processed based on the pronunciation attribute features and the target attribute features.

[0074] The above scheme extracts the pronunciation attribute features of the speech to be processed, and then obtains the target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes; finally, the synthesized speech of the speech to be processed is synthesized based on the pronunciation attribute features and the target attribute features. On the one hand, since there is no need for manual large-scale speech recording, time is saved, which helps to reduce the cost of speech synthesis. On the other hand, in the speech synthesis process, by obtaining the target categories of the speech to be processed on several speech attributes, the target attribute features of various speech attributes are obtained, and then speech synthesis is performed on this basis, so that speech data with any category of speech attributes can be synthesized, which improves the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0075] In some disclosed embodiments, the speech synthesis device 60 also includes a selection module for selecting at least one speech attribute as a first speech attribute and using unselected speech attributes as second speech attributes; the acquisition module 62 includes an acquisition submodule and an extraction submodule, the acquisition submodule is used to acquire target attribute features of each first speech attribute based on the target category of the speech to be processed on each first speech attribute; the extraction submodule is used to extract speech attribute features of the speech to be processed with respect to each second speech attribute as target attribute features corresponding to the second speech attributes.

[0076] Therefore, by migrating speech attribute features and completing speech attribute feature migration, a variety of synthesized speech with different categories of synthesized attribute features can be controlled.

[0077] See also Figure 7 , Figure 7 : is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 70 includes a memory 71 and a processor 72 coupled to each other, the memory 71 stores program instructions, and the processor 72 is used to execute the program instructions to implement the steps in any of the above-mentioned speech synthesis method embodiments. Specifically, the electronic device 70 may include but is not limited to: a desktop computer, a laptop computer, a server, a mobile phone, a tablet computer, etc., which are not limited here.

[0078] Specifically, the processor 72 is used to control itself and the memory 71 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 72 can also be called a CPU (Central Processing Unit). The processor 72 may be an integrated circuit chip with signal processing capabilities. The processor 72 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 72 can be implemented by an integrated circuit chip.

[0079] The above scheme, on the one hand, saves time and helps reduce the cost of speech synthesis because it does not require manual recording of a large number of voices. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized in several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis. It can synthesize speech data with any category of speech attributes, thereby improving the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0080] See also Figure 8 , Figure 8 The computer-readable storage medium 80 stores program instructions 81 that can be executed by a processor, and the program instructions 81 are used to implement the steps in any of the above-mentioned speech synthesis method embodiments.

[0081] The above scheme, on the one hand, saves time and helps reduce the cost of speech synthesis because it does not require manual recording of a large number of voices. On the other hand, in the speech synthesis process, by obtaining the target categories of the text to be synthesized in several speech attributes, the target attribute features of various speech attributes are obtained, and speech synthesis is performed on this basis. It can synthesize speech data with any category of speech attributes, thereby improving the freedom of speech synthesis. Therefore, it is possible to reduce its cost while improving the freedom of speech synthesis.

[0082] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0083] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0084] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0086] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor (processor) to perform all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A speech synthesis method, characterized in that: include: Extracting pronunciation attribute features of the text to be synthesized; Based on the target categories of the text to be synthesized on several speech attributes, respectively, obtaining target attribute features of various speech attributes; Based on the pronunciation attribute features and each of the target attribute features, synthesizing to obtain a synthesized speech of the text to be synthesized; Among them, the method of obtaining target attribute features of various speech attributes based on the target categories of the text to be synthesized on several speech attributes at least includes: performing feature modeling based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes, wherein the attribute space probability distribution model is used to divide different attribute categories into regions to obtain distribution spaces of different attribute categories under the same speech attribute; based on the target category of the text to be synthesized on the speech attribute, sampling is performed in the attribute space probability distribution model corresponding to the speech attribute to obtain target attribute features of the corresponding speech attribute.

2. The method according to claim 1, characterized in that The several speech attributes include: at least one of channel attributes, dialect attributes, timbre attributes, emotion attributes, and style attributes, wherein the channel attribute is used to characterize the environment in which the speech is located.

3. The method according to claim 1, characterized in that Each of the speech attributes has at least one of the attribute categories; the feature modeling is performed based on the speech data related to the attribute categories of the various speech attributes to obtain the attribute space probability distribution model of the various speech attributes, including: For each of the attribute categories under the speech attribute, extracting sample attribute features of the speech data related to the attribute category with respect to the speech attribute; Based on the sample attribute features of various attribute categories under the speech attributes, an attribute space probability distribution model of the speech attributes is constructed.

4. The method according to claim 1, characterized in that: The speech attribute includes a timbre attribute, and the step of acquiring the target attribute feature of the timbre attribute includes: Based on the target category of the to-be-synthesized text on the timbre attribute, obtaining a reference attribute feature of the timbre attribute; wherein the reference attribute feature of the timbre attribute is extracted based on speech data related to the target category of the timbre attribute; An adjustment process is performed based on the reference attribute feature of the timbre attribute to obtain a target attribute feature of the timbre attribute.

5. The method according to claim 4, characterized in that The reference attribute feature of the timbre attribute is adjusted to obtain the target attribute feature of the timbre attribute. Includes any of the following: Performing weighted processing on the plurality of reference attribute features of the timbre attribute to obtain a target attribute feature of the timbre attribute; The reference attribute feature of the timbre attribute is scaled to obtain the target attribute feature of the timbre attribute.

6. A speech synthesis method, characterized in that: include: Extracting pronunciation attribute features of the speech to be processed; Based on the target categories of the speech to be processed on several speech attributes respectively, obtaining target attribute features of various speech attributes; Based on the pronunciation attribute features and each of the target attribute features, synthesizing to obtain a synthesized speech of the speech to be processed; Among them, the target attribute features of various speech attributes are obtained based on the target categories of the speech to be processed on several speech attributes, which at least includes: feature modeling based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes, wherein the attribute space probability distribution model is used to divide different attribute categories into regions to obtain the distribution space of different attribute categories under the same speech attribute; based on the target category of the speech to be processed on the speech attribute, sampling is performed in the attribute space probability distribution model corresponding to the speech attribute to obtain the target attribute features of the speech attribute.

7. The method according to claim 6, characterized in that Before acquiring target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes, the method includes: Selecting at least one of the speech attributes as a first speech attribute, and using unselected speech attributes as second speech attributes; The step of obtaining target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes includes: Based on the target categories of the to-be-processed speech on the first speech attributes, respectively, obtaining target attribute features of the first speech attributes; and Speech attribute features of the speech to be processed related to each of the second speech attributes are extracted as target attribute features corresponding to the second speech attributes.

8. A speech synthesis device, characterized in that: include: An extraction module, used to extract the pronunciation attribute features of the text to be synthesized; An acquisition module is used to acquire target attribute features of various speech attributes based on the target categories of the text to be synthesized on several speech attributes, wherein the acquisition of target attribute features of various speech attributes based on the target categories of the text to be synthesized on several speech attributes at least includes: performing feature modeling based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes, wherein the attribute space probability distribution model is used to divide different attribute categories into regions to obtain distribution spaces of different attribute categories under the same speech attribute; based on the target category of the text to be synthesized on the speech attribute, sampling is performed in the attribute space probability distribution model corresponding to the speech attribute to obtain the target attribute features of the corresponding speech attribute; A synthesis module is used to synthesize the synthesized speech of the text to be synthesized based on the pronunciation attribute features and each of the target attribute features.

9. A speech synthesis device, characterized in that: include: An extraction module, used for extracting pronunciation attribute features of the speech to be processed; An acquisition module is used to acquire target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes, wherein the acquisition of target attribute features of various speech attributes based on the target categories of the speech to be processed on several speech attributes at least includes: performing feature modeling based on speech data related to the attribute categories of various speech attributes to obtain attribute space probability distribution models of various speech attributes, wherein the attribute space probability distribution model is used to divide different attribute categories into regions to obtain distribution spaces of different attribute categories under the same speech attribute; based on the target category of the speech to be processed on the speech attribute, sampling is performed in the attribute space probability distribution model corresponding to the speech attribute to obtain the target attribute features of the speech attribute; A synthesis module is used to synthesize the synthesized speech of the speech to be processed based on the pronunciation attribute features and each of the target attribute features.

10. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method according to any one of claims 1 to 7.

11. A computer-readable storage medium, characterized in that: Program instructions that can be executed by a processor are stored, and the program instructions are used to implement the speech synthesis method according to any one of claims 1 to 7.

Images