Speech synthesis method and device, equipment and storage medium

Abstract

The invention discloses a speech synthesis method and device, equipment and a storage medium. The method comprises the following steps: acquiring a coding feature corresponding to a target text; based on the target emotion type and the target emotion intensity coefficient, generating an emotion intensity feature; fusing the emotion intensity features and the coding features to generate acoustic representation; and based on the acoustic representation, generating voice data corresponding to the target text and having emotions. Through the method, the generated voice data can be more vivid.

Description

Technical Field

[0001] The present application relates to the field of speech processing, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art

[0002] Speech synthesis can convert text input information into understandable and fluent spoken speech data. However, although general speech synthesis methods can ensure accurate pronunciation, they are still insufficient in expressing the intensity of the real emotions of the speech. This process is often limited to simple text-to-speech conversion, ignoring the crucial emotional dimension of speech, making the generated speech sound lacking in liveliness. Summary of the invention

[0003] The main technical problem solved by the present application is to provide a speech synthesis method, apparatus, device and storage medium, which can make the generated speech data vivid.

[0004] To solve the above technical problems, a technical solution adopted in the present application is: to provide a speech synthesis method, the method comprising: obtaining coding features corresponding to the target text; generating emotion intensity features based on the target emotion type and the target emotion intensity coefficient; fusing the emotion intensity features and the coding features to generate acoustic representation; based on the acoustic representation, generating speech data corresponding to the target text and having emotion.

[0005] To solve the above technical problems, another technical solution adopted in the present application is: to provide an interactive device, including: an acquisition module, used to obtain the coding features corresponding to the target text; a feature generation module, used to generate emotion intensity features based on the target emotion type and the target emotion intensity coefficient; a representation generation module, used to fuse the emotion intensity features and the coding features to generate an acoustic representation; a processing module, based on the acoustic representation, generates speech data corresponding to the target text and with emotion.

[0006] In order to solve the above technical problems, another technical solution adopted in the present application is: to provide an electronic device, including a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the above speech synthesis method.

[0007] In order to solve the above technical problems, another technical solution adopted by the present application is: providing a computer-readable storage medium, which is used to store program instructions, and the program instructions can be executed to implement the above speech synthesis method.

[0008] The above scheme determines the emotion intensity characteristics through the emotion type and emotion intensity coefficient, and then integrates the emotion intensity characteristics and the encoding characteristics corresponding to the target text to obtain the acoustic representation. This process can integrate the emotion dimension and its intensity into the acoustic representation, and then generate speech data with the corresponding emotion dimension and its intensity based on the acoustic representation, making the generated speech data more vivid. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Figure 1 It is a flowchart of an embodiment of a speech synthesis method provided by the present application; Figure 2 It is a partial flow chart of another embodiment of the speech synthesis method provided by the present application; Figure 3 It is a schematic diagram of the overall process of the training process of the emotion feature extraction model provided by this application; Figure 4 It is a schematic diagram of an overall process involved in the generation of emotion intensity regulation features provided by the present application; Figure 5 It is a schematic diagram of the framework of an embodiment of a speech synthesis device of the present application; Figure 6 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 7 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0010] In order to make the purpose, technical solution and effect of the present application clearer and more specific, the present application is further described in detail below with reference to the accompanying drawings and examples.

[0011] It should be noted that the term "several" in this article means at least one. The term "and / or" is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the previous and subsequent associated objects are in an "or" relationship. In addition, "many" in this article means two or more than two. In addition, the term "at least one" in this article means any combination of at least two of any one or more of a plurality of, for example, including at least one of A, B, and C, can mean including any one or more elements selected from the set consisting of A, B, and C.

[0012] See also Figure 1 , Figure 1 1 is a flow chart of an embodiment of the speech synthesis method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not take Figure 1The method of this embodiment is executed by a speech synthesis device, which can be any device with processing capabilities, such as a computer, a mobile phone, a server, etc. Figure 1 As shown, this embodiment includes: Step S110: Obtain the encoding features corresponding to the target text.

[0013] The target text may be provided by the target user or determined by the speech synthesis device itself. Specifically, the speech synthesis device may be provided with a human-computer interaction interface, and the target text input by the user may be obtained through the human-computer interaction interface. Of course, in other embodiments, the speech synthesis device has a communication module, and the speech synthesis device receives the target text sent by the terminal device through the communication module. The terminal device has a human-computer interaction function, and the user uses the human-computer interaction function to input the target text to be sent to the speech synthesis device on the terminal device.

[0014] The coding features in this embodiment are features that convert the target text into numerical or vector features that can be understood and processed by a computer. Specifically, the coding features may include text features and sound-related features.

[0015] In some embodiments, the coding features include text features and sound-related features, and the sound-related features include at least one of phoneme features and audio features. Specifically, this step S110 may include: extracting text features from the target text; and extracting the phoneme information corresponding to the target text to obtain the phoneme features corresponding to the target text, or, obtaining predicted audio features based on the target text. Among them, the aforementioned coding features can be extracted by using an encoder of a pre-trained model. For example, the coding features of the target text are extracted using an encoder of an acoustic model. When the coding features include multiple, such as text features, phoneme features and audio features, the acoustic model may include multiple encoders for extracting the above-mentioned partial coding features, such as including a text encoder and a phoneme feature encoder, corresponding to the extraction of text features and phoneme features. Specifically, a text encoder is used to extract text features from a target text; phoneme information corresponding to the target text is obtained, and a phoneme encoder is used to perform feature extraction on the phoneme information corresponding to the target text to obtain phoneme features; audio features corresponding to the target text are predicted based on at least one of the text features and phoneme features of the target text, such as performing nonlinear transformation or multimodal fusion on the text features and phoneme features of the target text to predict the corresponding audio features.

[0016] In one embodiment, the target text may be encoded and feature extracted by using a pre-trained model or a feature extraction algorithm. For example, the encoding features corresponding to the target text are obtained through an acoustic model. The acoustic model needs to set the parameters of the initial model and define the structure required for the model before training. For example, a neural network is used to capture the mapping relationship between acoustic features and text alignment. A large number of training samples are obtained, and the training sample data is input into the model to obtain sample features. The model parameters are updated by continuously iterating the training through at least one of forward calculation and loss calculation, so that the model has good generalization ability and prediction accuracy after the training is completed. In a specific embodiment, the training process in the acoustic model includes the following steps: using multiple pre-stored audio data (which may come from multiple speakers and contain multiple emotion types) in the pre-stored sound library as training audio data, and obtaining the training text and training phoneme information corresponding to the training audio data, using the pre-trained text encoder to extract the sample text features of the training text; using the pre-trained phoneme feature encoder to extract the sample phoneme features of the training phoneme information; using the pre-trained audio feature extractor to extract the real audio features of the training audio data, or obtaining the audio features of the pre-extracted training audio data from the pre-stored sound library as the real audio data; using the aforementioned sample text features and sample phoneme features to predict the corresponding sample audio features through nonlinear transformation or multimodal fusion. Compare the predicted sample audio features with the real audio features, calculate the loss value and update the model parameters. When the loss value reaches convergence, the acoustic model can learn the mapping relationship from text features and phoneme features to audio features, that is, it has the ability to predict relatively accurate audio features through text features and phoneme features.

[0017] Step S120: Generate an emotion intensity feature based on the target emotion type and the target emotion intensity coefficient.

[0018] In this embodiment, at least one of the target emotion type and the target emotion intensity coefficient may be selected by the user.

[0019] In one embodiment, the emotion type and emotion intensity coefficient selected by the user are used as the target emotion type and target emotion intensity coefficient. The emotion intensity coefficient can be selected from a range of values ​​or a slide bar displayed on the configuration interface provided by the voice device, so that the user can select the target emotion intensity.

[0020] Specifically, based on the target emotion type, the emotion feature corresponding to the target emotion type can be obtained, and then the emotion feature corresponding to the target emotion type can be adjusted using the target emotion intensity coefficient to obtain the emotion intensity feature. Alternatively, the emotion feature corresponding to a reference emotion type can be obtained, and then the emotion intensity feature can be generated using the emotion feature corresponding to the reference emotion type and the feature difference between the reference emotion type and the target emotion type that matches the target emotion intensity coefficient. The reference emotion type can be, but is not limited to, a neutral emotion type, which is a neutral emotion without a specific emotion tendency, that is, a neutral emotion that is neither positive nor negative.

[0021] The emotional features corresponding to each emotion type described in this article can be obtained from a feature library, or can be obtained by extracting features from reference audio data of the corresponding emotion type using an emotion feature extraction model. The reference audio data can come from a cloud database or a local pre-stored sound library. The feature library is used to pre-store the emotional features corresponding to each emotion type. It can be understood that the features pre-stored in the feature library can also be extracted from audio data of the corresponding emotion type in advance using an emotion feature extraction model.

[0022] Step S130: Fusing the emotion intensity feature and the coding feature to generate an acoustic representation.

[0023] The acoustic representation may include information in multiple dimensions to ensure that the subsequently synthesized speech is vivid and natural. For example, the acoustic representation may include at least two of spectrum, fundamental frequency, audio, prosody, phoneme, and emotional features.

[0024] The corresponding emotion intensity features and coding features may be fused to generate acoustic representations by weighted fusion, concatenation or feature selection. For example, the emotion intensity features and coding features may be concatenated to obtain fused coding features, and then the fused coding features may be decoded to obtain acoustic representations.

[0025] In one embodiment, step S110 and step S130 can be implemented using the same pre-trained model. For example, step S110 is implemented by the encoder of the acoustic model, the fusion feature is obtained by fusing the emotion intensity feature and the coding feature in step S130, and then the decoder of the acoustic model decodes the fusion feature to obtain the acoustic representation. Among them, the decoder can be a decoder of a conventional convolution and attention structure. For example, the encoder of the acoustic model obtains the coding feature corresponding to the target text; the emotion intensity feature and the coding feature are spliced ​​to obtain the fused coding feature. Specifically, when the coding feature includes text features and sound-related features, the text features, sound-related features and emotion intensity features are spliced ​​to obtain the fused coding feature. The sound-related features may include at least one of the phoneme features and the predicted audio features; the fused coding feature is decoded by the decoder of the acoustic model to obtain the acoustic representation.

[0026] Step S140: Generate speech data corresponding to the target text and having emotion based on the acoustic representation.

[0027] In one embodiment, a vocoder may be used to generate speech data. For example, the vocoder may be a pre-trained general vocoder. The vocoder synthesizes an audio waveform based on an acoustic representation, and finally obtains speech data synthesized by emotional language. Specifically, the vocoder may be a vocoder for a specific language, and is used to generate speech data for a specific language based on the vocoder representation. The specific language may be, for example, but not limited to, English, Japanese, Chinese, etc.

[0028] This embodiment determines the emotion intensity feature through the emotion type and emotion intensity coefficient, and then integrates the emotion intensity feature and the encoding feature corresponding to the target text to obtain the acoustic representation. This process can integrate the emotion dimension and its intensity into the acoustic representation, and then generate speech data with the corresponding emotion dimension and its intensity based on the acoustic representation, making the generated speech data more vivid. Moreover, by integrating the emotion intensity into the acoustic representation, the generation of speech data corresponding to the emotion intensity can be realized, thus realizing the fine-grained emotion embodiment of the speech data.

[0029] In another embodiment, step S120 and step S130 are performed when the target user selects the target emotion intensity coefficient. When the target user does not select the target emotion intensity coefficient, before step S140, it may further include: obtaining general emotion features based on the target emotion type; fusing the general emotion features and the coding features to generate an acoustic representation. Thus, in step S140, voice data is generated using the acoustic representation based on the general emotion features. Among them, the general emotion features can be understood as the emotion features corresponding to the target emotion type under the default emotion intensity. Among them, the fusion method of the general emotion features and the coding features can refer to the relevant description of the fusion method of the emotion intensity features and the coding features in step S130.

[0030] The aforementioned universal emotional feature can be specifically determined by at least one emotional feature corresponding to the target emotional type. Each emotional feature of the target emotional type is extracted from a reference audio data belonging to the target emotional type, so the specific emotional feature acquisition step can refer to the relevant description of the following step S221. In a specific embodiment, an emotional feature corresponding to the target emotional type can be obtained as a universal emotional feature, or a plurality of emotional features corresponding to the target emotional type can be statistically analyzed to obtain a universal emotional feature that characterizes the central trend of multiple emotional features. For example, reference audio data of the same emotional type corresponding to different speakers are randomly sampled, and the plurality of reference audio data obtained by random sampling are respectively input into the emotional feature extraction model to obtain the corresponding emotional features, and the universal emotional feature is obtained by arithmetic averaging the obtained plurality of emotional features.

[0031] See also Figure 2 , Figure 2 FIG. 1 is a partial flow chart of another embodiment of the speech synthesis method provided by the present application. It should be noted that if there are substantially the same results, this embodiment does not necessarily refer to the embodiment of the present invention. Figure 2 The process sequence shown is limited. Figure 2 As shown, the above step S120 specifically includes: Step S221: Obtain representative emotion features of the benchmark emotion type.

[0032] The reference emotion type may be any emotion type used as a reference, such as a neutral emotion type, that is, without any specific emotion.

[0033] The representative emotional feature is determined based on at least one emotional feature corresponding to the benchmark emotional type. For example, an emotional feature corresponding to the benchmark emotional type is obtained as the representative emotional feature of the benchmark emotional type. Alternatively, multiple emotional features corresponding to the benchmark emotional type are statistically analyzed to obtain a representative emotional feature that characterizes the central trend of the multiple emotional features. Specifically, the multiple emotional features corresponding to the benchmark emotional type are averaged, mean or median to obtain the representative emotional feature of the benchmark emotional type. Each emotional feature of the benchmark emotional type is extracted from a reference audio data belonging to the benchmark emotional type, wherein the reference audio data can be obtained from a pre-stored sound library.

[0034] Emotional features can be obtained through machine learning models or pre-trained models. For example, emotional features are obtained by extracting features from reference audio data belonging to corresponding emotion types using an emotional feature extraction model. The emotional feature extraction model is trained using multiple sample audio data, which come from multiple speakers and belong to multiple emotion types. The sample audio data can be obtained from a pre-stored audio library.

[0035] In one embodiment, the step of extracting features from reference audio data using an emotional feature extraction model includes: extracting time series features from reference audio data using a time series feature extraction network of the emotional feature extraction model to obtain time series features. Mapping the time series features to a preset length feature dimension using the fully connected layer of the emotional feature extraction model. Extracting the mapped time series features using the emotional feature extraction module of the emotional feature extraction model to obtain emotional features. Among them, the time series feature extraction network includes at least one neural network. In a specific application scenario, a time series feature network is constructed using a multi-layer convolutional neural network and a recurrent neural network. Specifically, the reference audio data is first feature compressed by a multi-layer convolutional neural network, and then time series feature extraction is performed by a recurrent neural network to obtain the time series features corresponding to the reference audio data. Considering that the duration of different audios is different, in order to extract emotional features of uniform length, the emotional feature extraction model adds an additional fully connected layer to map hidden layer representations of different lengths to the same length feature dimension, that is, the time series features of reference audio data of different durations are all mapped to a uniform preset length feature dimension. The emotion feature extraction module is then used to extract the emotion features of fixed length corresponding to the reference audio data based on the mapped time series features.

[0036] In another embodiment, the emotional features obtained by extracting features from reference audio data using an emotional feature extraction model include at least one of the characteristics of enhanced emotional representation ability and decoupled speaker information. For example, in the training process of the emotional feature extraction model, it includes: the emotional feature extraction model extracts features from multiple sample audio data respectively to obtain sample emotional features corresponding to each sample audio data. The sample emotional features of each sample audio data are classified using at least one target classifier to obtain the classification results of each sample emotional feature by each target classifier. Based on the classification results corresponding to each target classifier and the corresponding type label, the classification loss of each target classifier is obtained. Based on the classification loss of each target classifier, the network parameters of the emotional feature extraction model are adjusted. Among them, at least one target classifier includes at least one of an emotional classifier and a speaker classifier. The difference between the classification result corresponding to the emotional classifier and the corresponding category label is the emotional classification difference, and the emotional classification difference is positively correlated with the classification loss of the emotional classifier. Thus, by minimizing the classification loss of the emotional classifier, the emotional features extracted by the emotional feature extraction model have enhanced emotional representation ability. The difference between the classification result corresponding to the speaker classifier and the corresponding category label is the speaker classification difference, which is negatively correlated with the classification loss of the speaker classifier. Therefore, by minimizing the classification loss of the classifier, it is difficult to determine the speaker of the emotion feature extracted based on the emotion feature extraction model, that is, the emotion feature extracted based on the emotion feature extraction model is decoupled from the speaker information of the reference audio data to obtain a purer emotion feature. Among them, the acquisition of sample emotion features can refer to the relevant description of feature extraction of reference audio data using the emotion feature extraction model in this step.

[0037] It can be understood that the aforementioned sample audio data and reference audio data may be audio data in the same pre-stored audio.

[0038] Combination Figure 3 To illustrate with an example, Figure 3It is a schematic diagram of the overall process of the training process of the emotion feature extraction model provided by the present application. Among them, the training process of the emotion feature extraction model includes: obtaining multiple sample audio data of multiple people and multiple emotions, that is, the multiple sample audio data come from multiple speakers and contain multiple emotion types, and the multiple emotion types include neutral emotion types. For each sample audio data, the sample audio data is first feature compressed by a convolutional neural network, and then the time series feature is extracted by a recurrent neural network to obtain the time series feature corresponding to the sample audio data. Using a fully connected layer, the time series feature corresponding to the sample audio data is mapped to a unified preset length feature dimension. The emotion feature extraction module is used to extract the emotion feature of a fixed length corresponding to the sample audio data based on the mapped time series feature, which is used as the sample emotion feature of the sample audio data of each emotion type output by the emotion feature extraction model. The output sample emotion feature is input into an emotion classifier composed of multiple layers of fully connected layers, and the emotion classifier is used to predict the emotion category to obtain the classification result corresponding to the emotion classifier, that is, the predicted emotion type label corresponding to the emotion feature. Based on the predicted sentiment type label With real emotion type label The difference between them is used to calculate the loss value, and the classification loss of the sentiment classifier is obtained. The classification loss of this sentiment classifier is Specifically, it can be the cross entropy loss ( Loss, CE Loss), which is calculated as follows: By minimizing This method makes the predicted emotion type label closer and closer to the real emotion type label. It is positively correlated with the classification loss of sentiment classification difference. In the case of the smallest value, the predicted emotion type label is closest to the real emotion type predicted label, that is, the emotion feature output by the emotion feature extraction model has enhanced emotion representation ability. In addition, the output sample emotion feature is input into a speaker classifier with multiple layers of fully connected layers of the same structure, and the speaker classifier is used to predict the speaker category, and the classification result corresponding to the speaker classifier is obtained, that is, the predicted speaker category label corresponding to the emotion feature. . The speaker category label will be predicted and real speaker category labels Calculate the loss value and get the classification loss of the speaker classifier Among them, a gradient reversal layer (GRL) is set between the emotion feature extraction model and the speaker category classifier, which can perform gradient reversal calculation on the loss value. Specifically, the classification loss of the emotion classifier is Specifically, it can be CE Loss, which is calculated as follows: .because Negatively correlated with the classification loss of speaker classification differences, The way in which the predicted speaker label is getting closer and closer to the true speaker category label. Understandably, In the smallest case, the predicted speaker category label is least close to the real speaker category label, that is, the emotional features output by the emotional feature extraction model are decoupled from the speaker information. By continuously iterating and adjusting the parameters in the emotional feature extraction model, the sample emotional features extracted by the emotional feature extraction model from the sample audio data can have emotional representation and reduce the dependence on the speaker information. That is, the emotional features extracted by the trained emotional feature extraction model from the reference audio data can have emotional representation and reduce the dependence on the speaker information.

[0039] Step S222: Obtain the emotional feature difference that matches the target emotional intensity coefficient.

[0040] The emotional feature difference represents the feature difference between the reference emotional type and the target emotional type. At least one of the target emotional type and the target emotional intensity coefficient may be selected by the target user.

[0041] In one embodiment, obtaining an emotional feature difference that matches a target emotional intensity coefficient includes: obtaining an emotional change vector corresponding to a target emotional type; adjusting the emotional change vector using the target emotional intensity coefficient to obtain an emotional feature difference. The emotional change vector represents the direction of emotional change from a baseline emotional type to a target emotional type. The above step of adjusting the emotional change vector using the target emotional intensity coefficient to obtain an emotional feature difference may specifically be to multiply the emotional change vector by the target emotional intensity coefficient.

[0042] The emotion change vector in this embodiment can be obtained in advance or in real time through a supervised learning model. For example, it is obtained through a support vector binary classifier. In a specific implementation, a number of emotion features corresponding to the target emotion type are used as positive samples, and a number of emotion features corresponding to the baseline emotion type are used as negative samples, and the support vector binary classifier corresponding to the target emotion type is obtained by training with positive samples and negative samples. A hyperplane used for binary classification is obtained from the support vector binary classifier corresponding to the target emotion type; a normal vector perpendicular to the hyperplane is obtained as the emotion change vector corresponding to the target emotion type. Among them, each emotion feature corresponding to the target emotion type is extracted from a reference audio data belonging to the target emotion type, and the specific instructions for obtaining the emotion feature can refer to the relevant description in step S221. The audio data corresponding to the positive sample and the audio data corresponding to the negative sample come from the same speaker or different speakers.

[0043] In this embodiment, the audio data corresponding to the positive sample and the audio data corresponding to the negative sample come from the same speaker, wherein the speaker category is selected by the target user or automatically recommended by the system.

[0044] In another embodiment, the emotion intensity coefficient can be selected from a range of values ​​displayed on the configuration interface provided by the voice device to obtain different emotion feature differences, and the user can select the emotion intensity. For example, the emotion intensity coefficient can be any decimal or selectable number between 0 and 1, and the user can select different emotion intensity coefficients to combine with the emotion change vector to generate emotion feature differences with different emotion intensities. In a specific embodiment, the value of the emotion intensity coefficient is divided into 20 levels, that is, the value 0-1 is divided into 20 equal parts corresponding to different emotion intensities for the user to choose.

[0045] In this embodiment, the audio data of a specific emotion type and a reference emotion type are subjected to a support vector machine binary classification model of the specific emotion type to obtain the emotion change vector corresponding to the specific emotion type, and then fine-grained emotion intensity adjustment is achieved based on the representative emotion features of the initial reference emotion type, the emotion change vector of the specific emotion type, and the input emotion intensity coefficient. Compared with linear interpolation emotion intensity adjustment, the emotion intensity adjustment scheme of this embodiment will not cause a decrease in the quality of the synthesized sound due to the destruction of the speaker information, and also supports more fine-grained smooth emotion intensity adjustment.

[0046] Step S223: Obtain the emotion intensity feature based on the representative emotion feature and the emotion feature difference.

[0047] In one implementation, the representative emotion feature and the emotion feature difference may be added to obtain the emotion intensity feature.

[0048] Combine the following Figure 4 To illustrate with an example, Figure 4This is a schematic diagram of the overall process of generating an emotion intensity adjustment feature provided by the present application. The overall process of generating an emotion intensity adjustment feature includes: using pre-stored audio data in a pre-stored sound library to extract emotion features through a trained emotion feature extraction model. And classify the emotion features of different speakers according to emotion categories. Use the emotion features corresponding to multiple audio data of the target emotion type that needs to be adjusted for emotion intensity as positive samples, and the emotion features corresponding to multiple audio data of the neutral emotion type as negative samples, and train a support vector binary classifier for the target emotion type. Among them, the multiple audio data of the target emotion type and the multiple audio data of the neutral emotion type can come from the same speaker or multiple speakers. For example, audio data of the target emotion type and the neutral emotion type corresponding to multiple speakers are obtained from the pre-stored sound library. Extract the hyperplane used for binary classification from the trained support vector binary classifier, and calculate the normal vector n perpendicular to the hyperplane. In addition, perform arithmetic averaging on the emotion features of all or part of the audio data of the neutral emotion type in the pre-stored sound library to obtain a representative of the neutral emotion type. . After obtaining the target sentiment intensity coefficient input by the user after, It can be any decimal between 0 and 1, and the emotional intensity characteristics of the target emotional type corresponding to the target emotional intensity coefficient are calculated. .

[0049] In a specific application scenario, the models used in the relevant steps of the above embodiments, such as the acoustic model, the emotion feature extraction model, the support vector binary classification and the vocoder, etc., can be obtained by pre-training. Therefore, the following data preparation is performed before pre-training.

[0050] Step 1: Construct a pre-stored sound library. The pre-stored sound library stores audio data of multiple speakers and multiple emotion types and their corresponding texts. The audio data in the pre-stored sound library is provided with speaker category labels and emotion type labels. The audio data can be obtained from at least one of internal data or open source data, or directly recorded and collected through an audio acquisition device. Multiple emotion types include emotion types with specific emotions (such as happiness, anger, sadness, etc.) and emotion types without specific emotions. Usually, the number of speakers in the pre-stored sound library can be as large as possible, and the emotion types can be kept as consistent as possible.

[0051] Step 2: Preprocess the audio data to reduce the impact of the audio data on model training. For example, energy regularization is performed on the audio data so that the audio data is adjusted to the same audio dimension, thereby fully capturing the key information in the audio data. The speaker category labels and emotion type labels corresponding to the audio data can also be automatically, semi-automatically or manually checked to ensure that there are no obvious differences in the expression of emotions. For example, manual screening is performed. Specifically, a sample check is performed on the sound library data to ensure that there are no obvious differences in the expression of emotions.

[0052] In a specific implementation, manual screening of audio data may include sampling and checking audio data of the same emotion type from different speakers to ensure that there is no significant difference in the emotion expression methods of audio data of the same emotion type from different speakers.

[0053] In another specific implementation, the emotion types corresponding to the audio data of different speakers should be consistent. The consistency here refers to the consistency in the number of emotion types corresponding to all audio data of different speakers.

[0054] Step 3: Obtain the feature data of each audio data and the corresponding text in the pre-stored sound library. For example, according to the pronunciation rules and the phoneme system, the text can be converted into corresponding phoneme information, rhythm information, etc., which can be used as input data for the text feature encoder for subsequent acoustic model training. In addition, the audio features of the audio data (which can be spectral features) can be extracted and used as the audio data required for subsequent acoustic model training.

[0055] See also Figure 5 , Figure 5 It is a framework diagram of an embodiment of a speech synthesis device of the present application. The speech synthesis device 500 includes a coding feature acquisition module 510, an emotion intensity feature generation module 520, a representation generation module 530 and a processing module 540. The coding feature acquisition module 510 is used to obtain coding features corresponding to the target text. The emotion intensity feature generation module 520 is used to generate emotion intensity features based on the target emotion type and the target emotion intensity coefficient. The representation generation module 530 is used to fuse the emotion intensity features and the coding features to generate an acoustic representation. The processing module 540 is used to generate speech data corresponding to the target text and having emotion based on the acoustic representation.

[0056] In some embodiments, the emotion intensity feature generation module 520 is specifically used to obtain representative emotion features of a baseline emotion type, wherein the representative emotion features are determined based on at least one emotion feature corresponding to the baseline emotion type, and each emotion feature of the baseline emotion type is extracted from a reference audio data belonging to the baseline emotion type; obtain emotion feature differences that match the target emotion intensity coefficient, and the emotion feature differences represent the feature differences between the baseline emotion type and the target emotion type; and obtain emotion intensity features based on the representative emotion features and the emotion feature differences.

[0057] In some embodiments, the emotion intensity feature generation module 520 executes to obtain the emotion feature difference that matches the target emotion intensity coefficient, including: obtaining the emotion change vector corresponding to the target emotion type, the emotion change vector representing the emotion change direction from the baseline emotion type to the target emotion type; using the target emotion intensity coefficient to adjust the emotion change vector to obtain the emotion feature difference.

[0058] In some embodiments, the emotion intensity feature generation module 520 also includes an emotion change vector acquisition module, which is used to obtain the emotion change vector corresponding to the target emotion type, including: taking a number of emotion features corresponding to the target emotion type as positive samples and a number of emotion features corresponding to the baseline emotion type as negative samples, and using the positive samples and negative samples to train a support vector binary classifier corresponding to the target emotion type, wherein each emotion feature corresponding to the target emotion type is extracted from a reference audio data belonging to the target emotion type, and the audio data corresponding to the positive sample and the audio data corresponding to the negative sample are from the same speaker or different speakers; obtaining a hyperplane used for binary classification from the support vector binary classifier corresponding to the target emotion type; and obtaining the normal vector of the vertical hyperplane as the emotion change vector corresponding to the target emotion type.

[0059] In some embodiments, the emotion intensity feature generation module 520 also includes an emotion feature difference acquisition module. The emotion feature difference acquisition module adjusts the emotion change vector using the target emotion intensity coefficient to obtain the emotion feature difference, including: multiplying the emotion change vector and the target emotion intensity coefficient to obtain the emotion feature difference.

[0060] In some embodiments, the emotion intensity feature generation module 520, when executing the process of obtaining representative emotion features of a baseline emotion type, includes: obtaining an emotion feature corresponding to the baseline emotion type as the representative emotion feature; or, performing statistics on multiple emotion features corresponding to the baseline emotion type to obtain a representative emotion feature that characterizes the central trend of multiple emotion features.

[0061] In some embodiments, the emotion intensity feature generation module 520 obtains the emotion intensity feature based on the representative emotion feature and the emotion feature difference, including: adding the representative emotion feature to the emotion feature difference to obtain the emotion intensity feature.

[0062] In some embodiments, the step of generating an emotion intensity feature based on the target emotion type and the target emotion intensity coefficient, and the step of generating an acoustic representation by fusing the emotion intensity feature and the coding feature are performed when the target user selects the target emotion intensity coefficient; the speech synthesis device 500 also includes a general emotion feature acquisition module 550, and before the processing module 540 generates speech data corresponding to the target text and having emotion based on the acoustic representation, it also includes: the general emotion feature acquisition module 550 is used to obtain a general emotion feature of the target emotion type based on at least one emotion feature corresponding to the target emotion type in response to the target user not selecting the target emotion intensity coefficient, wherein each emotion feature of the target emotion type is extracted from a reference audio data belonging to the target emotion type; the representation generation module 530 can also be used to fuse the general emotion feature and the coding feature to generate an acoustic representation.

[0063] In some embodiments, the emotion feature is obtained by extracting features from reference audio data belonging to the corresponding emotion type using an emotion feature extraction model; the emotion feature extraction model is trained using multiple sample audio data, and the multiple sample audio data come from multiple speakers and belong to multiple emotion types.

[0064] In some embodiments, both the reference audio data and the sample audio data are audio data in a pre-stored sound library.

[0065] In some embodiments, the speech synthesis device 500 can also be used to train an emotion feature extraction model, and the training steps of the emotion feature extraction model include: the emotion feature extraction model extracts features from multiple sample audio data respectively to obtain sample emotion features corresponding to each sample audio data; uses at least one target classifier to classify the sample emotion features of each sample audio data respectively to obtain the classification results of each target classifier for each sample audio data; obtains the classification loss of each target classifier based on the classification results corresponding to each target classifier and the corresponding type label; adjusts the network parameters of the emotion feature extraction model based on the classification loss of each target classifier; wherein, at least one target classifier includes at least one of an emotion classifier and a speaker classifier, the difference between the classification result corresponding to the emotion classifier and the corresponding category label is the emotion classification difference, and the emotion classification difference is positively correlated with the classification loss of the emotion classifier, and the difference between the classification result corresponding to the speaker classifier and the corresponding category label is the speaker classification difference, and the speaker classification difference is negatively correlated with the classification loss of the speaker classifier.

[0066] In some embodiments, the emotion feature is obtained by extracting features from audio data belonging to the corresponding emotion type using an emotion feature extraction model; the emotion intensity feature generation module 520 and the general emotion feature acquisition module 550 perform the steps of extracting features from audio data using the emotion feature extraction model, including: extracting time series features from the audio data using the time series feature extraction network of the emotion feature extraction model to obtain time series features; mapping the time series features to a preset length feature dimension using the fully connected layer of the emotion feature extraction model; and extracting the mapped time series features using the emotion feature extraction module of the emotion feature extraction model to obtain emotion features.

[0067] In some embodiments, the coding features include text features and sound-related features, and the sound-related features include at least one of phoneme features and audio features; the coding feature acquisition module 510 is specifically used to extract text features from the target text; and, to extract the phoneme information corresponding to the target text to obtain the phoneme features corresponding to the target text.

[0068] In some embodiments, the encoding features include text features and sound-related features, and the sound-related features include at least one of phoneme features and audio features; the encoding feature acquisition module 510 is specifically used to extract text features from the target text; and, to extract audio features corresponding to the target text from the simulated audio data corresponding to the target text.

[0069] In some embodiments, the step of obtaining the coding features corresponding to the target text and the step of fusing the emotion intensity features and the coding features to generate acoustic representation coding features are implemented using an acoustic model; the coding feature acquisition module 510 is specifically used to extract the coding features corresponding to the target text using the encoder of the acoustic model; the representation generation module 530 is specifically used to splice the text features, sound-related features and emotion intensity features to obtain fused coding features; the fused coding features are decoded using the decoder of the acoustic model to obtain an acoustic representation.

[0070] In some embodiments, at least one of the target emotion type and the target emotion intensity coefficient is selected by a target user.

[0071] See also Figure 6 , Figure 6 6 is a schematic diagram of a framework of an embodiment of an electronic device of the present application. The electronic device 60 includes a memory 61 and a processor 62 coupled to each other, and the processor 62 is used to execute program instructions stored in the memory 61 to implement the steps in any of the above-mentioned speech synthesis method embodiments. In a specific implementation scenario, the electronic device 60 may include but is not limited to: a microcomputer, a server, and in addition, the electronic device 60 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0072] Specifically, the processor 62 is used to control itself and the memory 61 to implement the steps in any of the above-mentioned speech synthesis method embodiments. The processor 62 can also be called a CPU (Central Processing Unit). The processor 62 may be an integrated circuit chip with signal processing capabilities. The processor 62 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 62 can be implemented by an integrated circuit chip.

[0073] See also Figure 7 , Figure 7 The computer-readable storage medium 70 stores program instructions 71 that can be executed by a processor, and the program instructions 71 are used to implement the steps in any of the above-mentioned speech synthesis method embodiments.

[0074] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0075] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0076] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0077] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0078] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A speech synthesis method, characterized in that: include: Obtain the encoding features corresponding to the target text; Generate emotion intensity features based on the target emotion type and the target emotion intensity coefficient; fusing the emotion intensity feature and the encoding feature to generate an acoustic representation; Based on the acoustic representation, speech data corresponding to the target text and having emotion is generated.

2. The method according to claim 1, characterized in that The generating of the emotion intensity feature based on the target emotion type and the target emotion intensity coefficient includes: Acquire a representative emotion feature of a reference emotion type, wherein the representative emotion feature is determined based on at least one emotion feature corresponding to the reference emotion type, and each emotion feature of the reference emotion type is extracted from a reference audio data belonging to the reference emotion type; Acquire an emotion feature difference that matches the target emotion intensity coefficient, wherein the emotion feature difference represents a feature difference between the reference emotion type and the target emotion type; The emotion intensity feature is obtained based on the representative emotion feature and the emotion feature difference.

3. The method according to claim 2, characterized in that The obtaining of the emotional feature difference matching the target emotional intensity coefficient includes: Acquire an emotion change vector corresponding to the target emotion type, wherein the emotion change vector represents an emotion change direction from the reference emotion type to the target emotion type; The target emotion intensity coefficient is used to adjust the emotion change vector to obtain the emotion feature difference.

4. The method according to claim 3, characterized in that The obtaining of the emotion change vector corresponding to the target emotion type includes: A plurality of emotional features corresponding to the target emotional type are used as positive samples, and a plurality of emotional features corresponding to the reference emotional type are used as negative samples, and a support vector binary classifier corresponding to the target emotional type is obtained by training using the positive samples and the negative samples, wherein each emotional feature corresponding to the target emotional type is extracted from a reference audio data belonging to the target emotional type, and the audio data corresponding to the positive samples and the audio data corresponding to the negative samples are from the same speaker or different speakers; Obtaining a hyperplane used for binary classification from the support vector binary classifier corresponding to the target emotion type; Obtaining a normal vector perpendicular to the hyperplane as an emotion change vector corresponding to the target emotion type; And / or, using the target emotion intensity coefficient to adjust the emotion change vector to obtain the emotion feature difference includes: The emotion change vector and the target emotion intensity coefficient are multiplied to obtain the emotion feature difference.

5. The method according to claim 2, characterized in that: The step of obtaining representative emotion features of the benchmark emotion type includes: Acquire an emotional feature corresponding to the reference emotional type as the representative emotional feature, or perform statistics on multiple emotional features corresponding to the reference emotional type to obtain a representative emotional feature that represents a central trend of the multiple emotional features; And / or, obtaining the emotion intensity feature based on the representative emotion feature and the emotion feature difference includes: The representative emotion feature and the emotion feature difference are added to obtain the emotion intensity feature.

6. The method according to claim 1, characterized in that The step of generating the emotion intensity feature based on the target emotion type and the target emotion intensity coefficient, and the step of fusing the emotion intensity feature and the coding feature to generate the acoustic representation are performed when the target user selects the target emotion intensity coefficient; Before generating the speech data corresponding to the target text and having emotion based on the acoustic representation, the method further includes: In response to the target user not selecting the target emotion intensity coefficient, obtaining a common emotion feature of the target emotion type based on at least one emotion feature corresponding to the target emotion type, wherein each emotion feature of the target emotion type is extracted from a reference audio data belonging to the target emotion type; The general emotion feature and the encoding feature are fused to generate an acoustic representation.

7. The method according to claim 2 or 6, characterized in that: The emotional features are obtained by extracting features from reference audio data belonging to the corresponding emotional type using an emotional feature extraction model; The emotion feature extraction model is trained using a plurality of sample audio data, where the plurality of sample audio data come from a plurality of speakers and belong to a plurality of emotion types.

8. The method according to claim 7, characterized in that The reference audio data and the sample audio data are both audio data in a pre-stored sound library; And / or, the method further comprises the following step of training the emotion feature extraction model: The emotion feature extraction model extracts features from the plurality of sample audio data respectively to obtain sample emotion features corresponding to each of the sample audio data; Using at least one target classifier to classify the sample emotion features of each sample audio data, respectively, to obtain classification results of each target classifier for each sample emotion feature; Obtaining the classification loss of each target classifier based on the classification result and the corresponding type label corresponding to each target classifier; Adjusting the network parameters of the emotion feature extraction model based on the classification loss of each target classifier; Among them, the at least one target classifier includes at least one of a sentiment classifier and a speaker classifier, the difference between the classification result corresponding to the sentiment classifier and the corresponding category label is the sentiment classification difference, and the sentiment classification difference is positively correlated with the classification loss of the sentiment classifier, and the difference between the classification result corresponding to the speaker classifier and the corresponding category label is the speaker classification difference, and the speaker classification difference is negatively correlated with the classification loss of the speaker classifier.

9. The method according to claim 2 or 6, characterized in that: The emotional features are obtained by extracting features from audio data belonging to the corresponding emotional type using an emotional feature extraction model; The step of extracting features from the audio data using the emotion feature extraction model comprises: Using the time series feature extraction network of the emotion feature extraction model to extract time series features from the audio data, to obtain time series features; Mapping the time series features to a preset length feature dimension using a fully connected layer of the sentiment feature extraction model; The emotional feature extraction module of the emotional feature extraction model is used to extract the mapped time series features to obtain the emotional features.

10. The method according to claim 1, characterized in that The encoding features include text features and sound-related features, and the sound-related features include at least one of phoneme features and audio features; The step of obtaining the encoding features corresponding to the target text includes: Extracting text features from the target text; Furthermore, the phoneme information corresponding to the target text is extracted to obtain the phoneme features corresponding to the target text, and / or the audio features of the target text are predicted based on at least one of the text features and phoneme features corresponding to the target text.

11. The method according to claim 1, characterized in that: The step of obtaining the coding feature corresponding to the target text and the step of fusing the emotion intensity feature and the coding feature to generate the acoustic representation coding feature are implemented by using an acoustic model; The step of obtaining the encoding features corresponding to the target text includes: Extracting encoding features corresponding to the target text using the encoder of the acoustic model; The fusing the emotion intensity feature and the coding feature to generate an acoustic representation includes: The text features, the sound-related features and the emotion intensity features are concatenated to obtain a fused coding feature; The fused coding features are decoded using a decoder of the acoustic model to obtain an acoustic representation.

12. The method according to claim 1, characterized in that At least one of the target emotion type and the target emotion intensity coefficient is selected by a target user.

13. An interactive device, characterized in that: The device comprises: A coding feature acquisition module is used to obtain coding features corresponding to the target text; An emotion intensity feature generation module, used to generate an emotion intensity feature based on a target emotion type and a target emotion intensity coefficient; A representation generation module, used for fusing the emotion intensity feature and the coding feature to generate an acoustic representation; A processing module generates speech data corresponding to the target text and having emotion based on the acoustic representation.

14. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores program instructions, and the processor is used to execute the program instructions to implement the speech synthesis method according to any one of claims 1-12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program instructions, and the program instructions can be executed to implement the speech synthesis method according to any one of claims 1 to 12.

Images