A speech synthesis method and apparatus

Abstract

The present application relates to the technical field of speech synthesis, and in particular to a speech synthesis method and device, the method comprising: fixing parameters of a speech synthesis model, taking higher similarity between real speech of a target object and speech of the target object synthesized by the speech synthesis model as an optimization target, updating and iterating feature values of an initial speech feature vector to obtain a target speech feature vector of the target object; inputting the target speech feature vector and text to be synthesized into the speech synthesis model to synthesize target speech corresponding to the text to be synthesized. The present application can reduce training cost and shorten synthesis time by adjusting the feature values of the initial speech feature vector to synthesize the speech of the target object, and can efficiently adapt to speech synthesis tasks of different target objects.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a speech synthesis method and device. BACKGROUND

[0002] Speech synthesis (Text-to-Speech, TTS) is a technology of converting text information into speech information. In the prior art, when a TTS model is required to synthesize the speech of a specific object, speech feature vectors are first extracted from speech samples of the object, and then the parameters in the TTS model are fine-tuned based on the speech feature vectors.

[0003] However, in order to ensure the accuracy and naturalness of synthesized speech of different speaking objects, a large number of parameters in the TTS model need to be repeatedly trained, and a set of parameters of one model can only adapt to one target speaking object. This results in high training cost and long time consumption, and it is difficult to efficiently adapt to speech synthesis tasks of different speakers. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a speech synthesis method and device, which can reduce training cost, shorten time consumption, and efficiently adapt to speech synthesis tasks of different speakers.

[0005] In a first aspect, an embodiment of the present application provides a speech synthesis method, which comprises:

[0006] obtaining a to-be-synthesized text corresponding to a target object;

[0007] updating and iterating the feature values of an initial speech feature vector to obtain a target speech feature vector of the target object, with the similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model being higher as the optimization target, and the text corresponding to the real speech being the same as the text corresponding to the speech;

[0008] inputting the target speech feature vector and the to-be-synthesized text into the speech synthesis model to synthesize the target speech corresponding to the to-be-synthesized text.

[0009] In a possible implementation, the updating and iterating the feature values of an initial speech feature vector to obtain a target speech feature vector of the target object comprises:

[0010] obtaining a first speech sample of the target object and a text sample corresponding to the first speech sample;

[0011] for each updating and iteration of the feature values, inputting the text sample and the latest initial speech feature vector into the speech synthesis model to obtain synthesized speech;

[0012] calculating a loss value of the speech synthesis model according to the first speech sample and the synthesized speech; the smaller the loss value is, the higher the similarity between the first speech sample and the synthesized speech is;

[0013] fixing parameters of the speech synthesis model, taking the minimum loss value of the speech synthesis model as an optimization target, and updating feature values of the latest initial speech feature vector according to the loss value.

[0014] In a possible implementation, the initial speech feature vector is obtained by the following steps:

[0015] performing feature fusion on preset speech feature vectors of preset speech samples of at least one preset object to obtain a fusion speech feature vector; the target object is included in the preset object; the preset speech feature vectors and the fusion speech feature vector include first feature values of each preset speech feature dimension;

[0016] filtering at least one target speech feature dimension from all preset speech features according to weights of each preset speech feature dimension; the weight is used to measure a degree to which the preset speech feature dimension can reflect unique speech features of the target object;

[0017] extracting first feature values corresponding to the target speech feature dimension from the fusion speech feature vector to form the initial speech feature vector.

[0018] In a possible implementation, the weight of each preset speech feature dimension to the target object is determined by the following steps:

[0019] extracting a sample speech feature vector corresponding to a second speech sample of the target object; the sample speech feature vector includes second feature values of each preset speech feature dimension;

[0020] performing speech feature decomposition on each sample speech feature vector to obtain second feature values of each sample speech feature vector in each preset speech feature dimension;

[0021] for each preset speech feature dimension, calculating the weight of the preset speech feature dimension to the target object according to the second feature values of the preset speech feature dimension in each sample speech feature vector.

[0022] In a possible implementation, the calculating the weight of the preset speech feature dimension to the target object according to the second feature values of the preset speech feature dimension in each sample speech feature vector includes:

[0023] calculate a variance of all second feature values of the preset speech feature dimension, to obtain a weight of the preset speech feature dimension on the target object.

[0024] In a possible implementation, the filtering of the target speech feature dimension from all preset speech features according to the weight of each preset speech feature dimension on the target object comprises:

[0025] sorting the weights on the target object for all preset speech feature dimensions;

[0026] determining a preset number of preset speech feature dimensions with the largest weights as the target speech feature dimensions.

[0027] In a possible implementation, before the feature fusion of the preset speech feature vectors of the preset speech samples of the at least one preset object to obtain the fused speech feature vector, the method further comprises:

[0028] removing the preset speech feature vectors with a difference degree greater than a preset difference degree from other preset speech feature vectors to obtain a final preset speech feature vector.

[0029] In a possible implementation, the removing of the preset speech feature vectors with a difference degree greater than a preset difference degree from other preset speech feature vectors comprises:

[0030] determining whether there is a preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors;

[0031] if there is, for each preset speech feature vector, counting a number of other preset speech feature vectors with a difference degree greater than a preset difference degree from the preset speech feature vector to obtain a difference degree corresponding to the preset speech feature vector;

[0032] after removing the preset speech feature vector with the largest difference degree, continuing to determine whether there is a preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors until the difference degrees between the preset speech feature vectors are all less than the preset difference degree.

[0033] In a possible implementation, the inputting of the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize a target speech corresponding to the text to be synthesized comprises:

[0034] inputting the text to be synthesized into a text encoder in the speech synthesis model to obtain a text to be synthesized vector;

[0035] inputting the target speech feature vector into an attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector of an auxiliary speech feature dimension based on the to-be-synthesized text vector, the auxiliary speech feature dimension and the target speech feature dimension constituting all speech feature dimensions; the attention model aligns the to-be-synthesized text vector with the target speech feature vector and the auxiliary speech feature vector;

[0036] inputting the correspondence between the to-be-synthesized text vector and the target speech feature vector and the auxiliary speech feature vector obtained after alignment into a decoder in the speech synthesis model, to obtain the target speech of the to-be-synthesized text.

[0037] In a second aspect, the embodiments of the present application further provide a speech synthesis device, the device comprising:

[0038] an acquisition module configured to acquire to-be-synthesized text corresponding to a target object;

[0039] an updating module configured to fix parameters of a speech synthesis model, and perform updating iteration on feature values of an initial speech feature vector, so as to obtain a target speech feature vector of the target object, the optimization target being that the similarity between real speech of the target object and speech of the target object synthesized by the speech synthesis model is higher, and the text corresponding to the real speech being the same as the text corresponding to the speech.

[0040] a synthesis module configured to input the target speech feature vector and the to-be-synthesized text into the speech synthesis model, and synthesize target speech corresponding to the to-be-synthesized text.

[0041] In a possible implementation, the updating module is specifically configured to acquire a first speech sample of the target object and a text sample corresponding to the first speech sample; input the text sample and the latest initial speech feature vector into the speech synthesis model to obtain synthesized speech for each updating iteration of the feature values; calculate a loss value of the speech synthesis model according to the first speech sample and the synthesized speech; the smaller the loss value is, the higher the similarity between the first speech sample and the synthesized speech is; fix the parameters of the speech synthesis model, and perform updating on the feature values of the latest initial speech feature vector according to the loss value, the optimization target being that the loss value of the speech synthesis model is the smallest.

[0042] In a possible implementation, the updating module is further configured to:

[0043] perform feature fusion on preset voice feature vectors of preset voice samples of at least one preset object to obtain a fused voice feature vector; the preset object includes the target object; the preset voice feature vectors and the fused voice feature vector include first feature values of each preset voice feature dimension;

[0044] select at least one target voice feature dimension from all preset voice features according to weights of each preset voice feature dimension; the weights are used to measure degrees at which the preset voice feature dimensions can reflect unique voice features of the target object;

[0045] extract first feature values corresponding to the target voice feature dimensions from the fused voice feature vector to form an initial voice feature vector.

[0046] In a possible implementation, the updating module is further configured to:

[0047] extract sample voice feature vectors corresponding to second voice samples of the target object; the sample voice feature vectors include second feature values of each preset voice feature dimension;

[0048] perform voice feature decomposition on each sample voice feature vector to obtain second feature values of each sample voice feature vector in each preset voice feature dimension;

[0049] for each preset voice feature dimension, calculate a weight of the preset voice feature dimension for the target object according to the second feature values of the preset voice feature dimension in each sample voice feature vector.

[0050] In a possible implementation, the updating module is further configured to:

[0051] calculate a variance of all second feature values of the preset voice feature dimension to obtain the weight of the preset voice feature dimension for the target object.

[0052] In a possible implementation, the updating module is further configured to:

[0053] sort the weights for all preset voice feature dimensions for the target object;

[0054] determine a preset number of preset voice feature dimensions with the largest weights as the target voice feature dimensions.

[0055] In a possible implementation, before the updating module performs feature fusion on preset voice feature vectors of preset voice samples of at least one preset object to obtain a fused voice feature vector, the updating module is further configured to:

[0056] remove the preset voice feature vectors with a difference greater than the preset difference from other preset voice feature vectors to obtain final preset voice feature vectors.

[0057] In a possible implementation, the updating module is further configured to:

[0058] determine whether there is a preset voice feature vector with a difference greater than the preset difference from other preset voice feature vectors;

[0059] if there is, count the number of other preset voice feature vectors with a difference greater than the preset difference from the preset voice feature vector for each preset voice feature vector to obtain a difference degree corresponding to the preset voice feature vector;

[0060] remove the preset voice feature vector with the greatest difference degree, and continue to determine whether there is a preset voice feature vector with a difference greater than the preset difference from other preset voice feature vectors until the differences between the preset voice feature vectors are all less than the preset difference.

[0061] In a possible implementation, the synthesizing module is specifically configured to: input the text to be synthesized into a text encoder in the speech synthesis model to obtain a text vector to be synthesized; input the target voice feature vector into an attention model in the speech synthesis model, so that the attention model generates an auxiliary voice feature vector in an auxiliary voice feature dimension based on the text vector to be synthesized, the auxiliary voice feature dimension and the target voice feature dimension constitute all voice feature dimensions; the attention model aligns the text vector to be synthesized with the target voice feature vector and the auxiliary voice feature vector; input the correspondence between the text vector to be synthesized and the target voice feature vector and the auxiliary voice feature vector obtained after alignment into a decoder in the speech synthesis model to obtain the target voice of the text to be synthesized.

[0062] The embodiment of the present application provides a speech synthesis method and device, the method comprises the following steps: fixing the parameters of a speech synthesis model, taking the higher similarity between the real speech of a target object and the speech of the target object synthesized by the speech synthesis model as an optimization target, updating and iterating the feature values of an initial speech feature vector to obtain a target speech feature vector of the target object; inputting the target speech feature vector and to-be-synthesized text into the speech synthesis model to synthesize the target speech corresponding to the to-be-synthesized text. The present application introduces an initial speech feature vector with a small number of parameters, instead of training all parameters in the speech synthesis model once for each target object, and the speech feature vector can be used to represent the speech features of different speakers by adjusting the feature values of the initial speech feature vector, and then the speech of the target object is synthesized. The scheme of the present application can reduce the training cost of copying different speaker timbres, shorten the synthesis time, and efficiently adapt to the speech synthesis task of different target objects. BRIEF DESCRIPTION OF DRAWINGS

[0063] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0064] Figure 1 A flow chart of a speech synthesis method provided by the embodiment of the present application is shown;

[0065] Figure 2 An update iteration flow chart of the feature values of the initial speech feature vector provided by the embodiment of the present application is shown;

[0066] Figure 3 A flow chart of the speech synthesis model synthesizing the target speech provided by the embodiment of the present application is shown;

[0067] Figure 4 A structural schematic diagram of a speech synthesis device provided by the embodiment of the present application is shown;

[0068] Figure 5 A structural schematic diagram of an electronic device provided by the embodiment of the present application is shown. DETAILED DESCRIPTION

[0069] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for the embodiments of the present application to make a clear and complete description of the technical solutions of the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportion. The flowchart used in the present application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or one or more operations can be removed from the flowchart under the guidance of the content of the present application.

[0070] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0071] In order to enable those skilled in the art to use the content of the present application, the following implementation is given in combination with a specific application scenario "speech synthesis technology field". For those skilled in the art, the general principles defined herein can be applied to other embodiments and application scenarios without departing from the spirit and scope of the present application. Although the present application is mainly described around "speech synthesis technology field", it should be understood that this is only an exemplary embodiment, for example, it can also be applied in the field of song synthesis technology.

[0072] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0073] The following will be a detailed description of a speech synthesis method provided by the embodiments of the present application.

[0074] Referring to Figure 1 The flowchart of the speech synthesis method provided by the embodiments of the present application is shown in FIG. 1. The following will be a description of the exemplary steps of the embodiments of the present application:

[0075] S101, obtaining a to-be-synthesized text corresponding to a target object.

[0076] In this application's embodiments, the target object can be any object capable of emitting sound, such as a person, bird, or tiger. The text to be synthesized refers to the text that needs to be synthesized into speech, such as "hello".

[0077] S102. Fix the parameters of the speech synthesis model, and with the goal of increasing the similarity between the real speech of the target object and the speech synthesized by the speech synthesis model, update and iterate the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object.

[0078] In this embodiment, the target object's authentic speech is a natural speech sample actually emitted by the target object, and the text corresponding to the authentic speech is the same as the text corresponding to the speech synthesized by the speech synthesis model. The initial speech feature vector includes feature values ​​for at least one preset speech feature dimension. The preset speech feature dimension corresponding to the initial speech feature vector is a speech feature dimension capable of distinguishing the target speaker. For example, if the initial speech feature vector includes feature values ​​for timbre, prosody, and speech rate, then timbre, prosody, and speech rate are speech feature dimensions capable of distinguishing the target speaker. The speech synthesis model uses the initial speech feature vector or the target speech feature vector to synthesize the target object's speech. The higher the similarity between the target object's authentic speech and the speech synthesized by the speech synthesis model, the more capable the initial or target speech feature vector is of distinguishing the target speaker. The target speech feature vector is a speech feature vector capable of distinguishing the target speaker.

[0079] The preset speech feature dimensions corresponding to the initial speech feature vector are determined based on the degree to which each preset speech feature dimension can reflect the unique speech features of the target object. For example, in a preferred embodiment, the preset speech feature dimensions include speech-related representations such as timbre, prosody, speech rate, intonation, and pitch (frequency). All preset speech feature dimensions are sorted according to their degree of reflecting the unique speech features of the target object X, resulting in timbre > intonation > speech rate > prosody > pitch. The first three are taken as the preset speech feature dimensions corresponding to the initial speech feature vector. Therefore, the preset speech feature dimensions corresponding to the target object X include timbre, intonation, and speech rate.

[0080] In the prior art, the speech feature vector corresponding to the target object is usually extracted from a large number of speech samples of the target object, and then a large number of parameters in the speech synthesis model are fine-tuned based on the speech feature vector. In contrast, the number of parameters in the TTS model is much larger than the number of parameters in the initial speech feature vector. Therefore, compared with the conventional way of synthesizing the speech of the target object by fine-tuning the parameters in the TTS model, the way of synthesizing the speech of the target object by adjusting the feature values of the initial speech feature vector in the present application can reduce the training cost, shorten the synthesis time, and efficiently adapt to different speech synthesis tasks of different target objects.

[0081] Referring to Figure 2 The updating iteration flowchart of the feature values of the initial speech feature vector provided by the embodiment of the present application is shown in FIG. 2, and the specific steps are as follows:

[0082] S201, obtaining a first speech sample of a target object and a text sample corresponding to the first speech sample.

[0083] In the present application, the first speech sample is a natural speech sample actually uttered by the target object. For example, if the first speech sample is a natural speech of "eating snacks" uttered by the target object, the text sample corresponding to the first speech sample is "eating snacks".

[0084] S202, for each updating iteration of the feature values, inputting the text sample and the latest initial speech feature vector into the speech synthesis model to obtain a synthesized speech.

[0085] S203, calculating a loss value of the speech synthesis model according to the first speech sample and the synthesized speech.

[0086] In the present application, the loss value of the speech synthesis model is used to quantify the gap between the first speech sample and the synthesized speech. The smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech, which means that the latest initial speech feature vector is more capable of distinguishing the target speaker and the effect of synthesizing the speech of the target object is better.

[0087] The loss value of the speech synthesis model can be calculated by using loss functions such as L1 Loss, L2 Loss and STFT Loss.

[0088] S204, fixing the parameters of the speech synthesis model, taking the minimum loss value of the speech synthesis model as the optimization objective, and updating the feature values of the latest initial speech feature vector according to the loss value.

[0089] Here, the "initial speech feature vector" is a fixed-size container containing at least one preset speech feature dimension value. For different target objects, the preset speech feature dimensions and / or feature values in the initial speech feature vector are fine-tuned, and different preset speech feature dimension values are used to represent the speech features of different target objects. The "initial speech feature vector" replaces the positions of the audio encoder and the sound feature extraction model in the traditional TTS model. The parameter amount of the "initial speech feature vector" is significantly lower than that of the audio encoder and the sound feature extraction model. The application does not need to fine-tune the parameters of the attention model and the decoder in the traditional TTS model, but only needs to fine-tune the "initial speech feature vector" for different target objects. The amount of parameters that need to be fine-tuned is significantly reduced, and the training overhead is significantly reduced.

[0090] It should be noted that the application still needs to fine-tune different "initial speech feature vectors" for different target objects. The first speech sample for fine-tuning an "initial speech feature vector" needs to correspond to a relatively stable timbre performance of a fixed target object, so the similarity between each first speech sample needs to meet the preset similarity requirement.

[0091] In addition, the application obtains the initial speech feature vector by the following steps:

[0092] Step 1: Perform feature fusion on the preset speech feature vectors of the preset speech samples of at least one preset object to obtain a fused speech feature vector.

[0093] In the embodiment of the application, the preset objects include target objects. One preset object can have one or more preset speech feature vectors of preset speech samples. The preset speech sample is a natural speech sample actually emitted by the preset object, such as the natural speech "Have you eaten?" actually emitted by the target object. The preset speech feature vector is extracted by inputting the preset speech sample into a speech feature vector extraction model. The preset speech feature vector and the fused speech feature vector include the first feature value of each preset speech feature dimension. All preset speech feature vectors have the same dimension, which is convenient for subsequent processing.

[0094] For example, the speech vector extraction model can use models such as HuBERT, Wav2Vec series, or x-vectors.

[0095] For example, the preset speech feature dimensions include timbre, prosody, speech rate, intonation, and pitch (frequency), and the preset speech feature vector includes the first feature value of the timbre, the first feature value of the prosody, the first feature value of the speech rate, the first feature value of the intonation, and the first feature value of the pitch.

[0096] Here, through feature fusion, it can be ensured that the fused speech feature vector can represent a better representation space of the speech features of any preset object, and will not be biased towards any preset object. Through this step, the time cost of subsequent fine-tuning of the initial speech feature vector of the target object can be reduced.

[0097] The preset speech samples are obtained after preprocessing. For example, if some preset speech samples are too long, the preset speech samples can be divided according to the preset upper limit of the length of the preset speech samples, and the preset speech samples can also be preprocessed by audio noise reduction, audio enhancement, etc.

[0098] Specifically, the fusion can be performed by using an average feature method, specifically, the average of the preset speech feature vectors corresponding to the several preset speech samples of the at least one preset object is calculated to obtain a fused speech feature vector.

[0099] For example, assuming that the preset speech feature vector dimensions of the preset speech sample a and the preset speech sample b are aligned, the preset speech feature vector of the preset speech sample a of the preset object A is [3, 8, 7], and the preset speech feature vector of the preset speech sample b of the preset object B is [7, 4, 5], then the average of the preset speech feature vector of the preset speech sample a of the preset object A and the preset speech feature vector of the preset speech sample b of the preset object B is calculated to obtain a fused speech feature vector [5, 6, 6].

[0100] Step two, according to the weight of each preset speech feature dimension, at least one target speech feature dimension is selected from all preset speech features.

[0101] In the embodiments of the present application, the weight of each preset speech feature dimension is used to measure the degree to which the preset speech feature dimension can reflect the unique speech feature of the target object.

[0102] For example, the preset speech feature dimensions include common representations such as tone, prosody, speech rate, intonation, pitch (frequency), and loudness related to speaking, wherein the tone can include features such as fundamental frequency and formant, in addition to less common features such as harmonic-to-noise ratio, speaking style, and accent. Taking the common representation as an example, assuming that the representation weight of tone is 0.6, the representation weight of prosody is 0.9, the representation weight of speech rate is 0.2, the representation weight of intonation is 0.1, and the representation weight of pitch is 0.7, and the preset speech feature dimensions with a representation weight greater than 0.6 are taken as target speech feature dimensions, then the target speech feature dimensions include prosody and pitch, which can highlight the speech features of the target object while further reducing the parameter amount of the speech feature vector, thereby reducing the training overhead.

[0103] Specifically, according to the weight of each preset speech feature dimension to the target object, target speech feature dimensions are selected from all preset speech feature dimensions, including:

[0104] i. The weight of each preset speech feature dimension to the target object is sorted;

[0105] ii. The preset number of preset speech feature dimensions with the largest weight are determined as the target speech feature dimensions.

[0106] For example, according to the weight of each preset speech feature dimension, the pitch > tone > intonation > speed > rhythm, and the first three are taken as the preset speech feature dimensions corresponding to the initial speech feature vector. Therefore, the preset speech feature dimensions corresponding to the target object X include tone, intonation, and speed.

[0107] In addition, the weight of each preset speech feature dimension to the target object is determined by the following steps:

[0108] i. The sample speech feature vector corresponding to the second speech sample of the target object is extracted.

[0109] In the embodiments of the present application, the second speech sample is a natural speech sample actually emitted by the target object, such as the natural speech "good morning" actually emitted by the target object. The sample speech feature vector is extracted by inputting the second speech sample into a speech feature vector extraction model, which includes the second feature values of each preset speech feature dimension.

[0110] For example, the preset speech feature dimensions include tone, rhythm, speed, intonation, and pitch (frequency), and the parameters of the sample speech feature vector can represent the second feature values of the tone, rhythm, speed, intonation, and pitch of the target object.

[0111] ii. Speech feature decomposition is performed on each sample speech feature vector to obtain the second feature values of each sample speech feature vector in each preset speech feature dimension.

[0112] In the embodiments of the present application, by feature decomposition, the high-dimensional sample speech feature vector is mapped to a preset number of interpretable and controllable subspaces of speech feature dimensions, i.e., mapped to the second feature values in each preset speech feature dimension.

[0113] For example, the preset speech feature dimensions include timbre, prosody, speech rate, intonation, and pitch (frequency), the sample speech feature vector R is [3, 7, 0, 2, 7], speech feature decomposition is performed on the sample speech feature vector R to obtain a second feature value 3 of the sample speech feature vector R in the timbre, a second feature value 7 of the sample speech feature vector R in the prosody, a second feature value 0 of the sample speech feature vector R in the speech rate, a second feature value 2 of the sample speech feature vector R in the intonation, and a second feature value 7 of the sample speech feature vector R in the pitch.

[0114] It should be understood that the above example is only for convenience of understanding, and in fact, the feature vector in the present application expresses the sound characteristics of the speaker object through a lower-dimensional dense representation. The speech feature vector is usually a hidden vector or an embedding vector, which is an abstract representation encoded by a speech vector extraction model after learning a large number of samples. The numerical values of each dimension have no intuitive physical meaning.

[0115] iii. For each preset speech feature dimension, a weight of the preset speech feature dimension to the target object is calculated according to the second feature values of the preset speech feature dimension in the sample speech feature vectors.

[0116] In the embodiments of the present application, the variance of all the second feature values of the preset speech feature dimension is calculated to obtain the weight of the preset speech feature dimension to the target object.

[0117] Here, the role of variance in synthesizing the target object's speech in the preset synthesis model includes: (1) capturing the key speech features of the target object: the preset speech feature dimension with large variance represents that the variation of the sample speech feature vector in this direction is more significant, which can capture the main variability in the sample speech feature vector. In speech synthesis, the speech features of the target object (such as timbre, pitch, formant, etc.) are the key to distinguishing different objects. The preset speech feature dimension with large variance can better capture these key features, thus preserving the unique "personality" of the target object in synthesized speech. (2) Improving the naturalness of synthesized speech: the naturalness of synthesized speech is one of the important indicators to measure the performance of a speech synthesis system. Natural speech is more easily accepted by users, especially in scenarios that require long-term interaction (such as smart assistants, audiobooks, etc.). The preset speech feature dimension with large variance can preserve more features related to the natural speech of the target object, making the synthesized speech more similar to the target object's speech in terms of timbre, prosody, and intonation, thus improving the naturalness. (3) Enhancing speaker distinguishability: by preserving the preset speech feature dimension with large variance, the synthesized speech can better simulate the timbre and speech features of the target object, thus enhancing the distinguishability of the target object. In summary, in the field of speech synthesis, the preset speech feature dimension with large variance has significant advantages in synthesizing the speech of the target object. They can better capture the key speech features of the target object, improve the naturalness and distinguishability of the synthesized speech of the target object.

[0118] Step three, extracting the first feature value corresponding to the target speech feature dimension from the fusion speech feature vector to form an initial speech feature vector.

[0119] For example, the preset speech feature dimensions include timbre, prosody, speech rate, intonation, and pitch (frequency), the fusion speech feature vector is [3, 7, 9, 2, 8], and the target speech feature dimension is prosody and speech rate. Then the initial speech feature vector is [7, 9].

[0120] In the embodiments of the present application, different first difference degree parameters can be set for different voice samples of different target objects, and the setting of the first difference degree parameter is related to the particularity (or scarcity) of the preset voice feature vector of the preset voice sample of the preset object, the audio quality of the preset voice sample, and the like. For example, for an object with a more particular accent, or a dialect or language with a very small number of users, the preset initialization difference degree parameter can be a larger value, while for a common language, the preset initialization difference degree can be a smaller value. If the audio quality is high, such as a higher sampling rate, sampling depth, or bit rate, the preset initialization difference degree parameter can be smaller. In a preferred embodiment, for a target object, the determination method of the first difference degree parameter can include: selecting a voice feature vector corresponding to a voice sample with the highest audio quality as a reference vector, and taking the audio quality parameter as the horizontal axis, for example, selecting the preset voice feature vector corresponding to the voice sample with the highest sampling rate as the reference vector, then taking the sampling rate parameter as the horizontal axis, and taking the spatial distance between the preset voice feature vector extracted from other voice samples and the reference vector as the vertical axis to draw a scatter plot, and based on the scatter plot, determining a threshold point, for example, if the spatial distance corresponding to 98% of the voice samples is below 0.1, then 0.1 is taken as the 98th percentile point as the first difference degree parameter of the target object. When training for the same target object, only voice samples meeting the first difference degree parameter are used for training, which further reduces the training overhead for each target object while highlighting the voice features of the target object.

[0121] It should be understood that when performing voice synthesis using the scheme of the present application, for different target objects, the initial voice feature vector still needs to be trained based on the voice samples of each target object to obtain a target voice feature vector corresponding to each target object.

[0122] Further, before the feature fusion is performed on the preset speech feature vectors of the preset speech samples of the at least one preset object to obtain the fused speech feature vector, the method further includes: removing the preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors to obtain a final preset speech feature vector. If the difference degrees between the preset speech feature vectors are relatively large, the fused speech feature vector may be biased towards part of the preset objects, so that the initial speech feature vector is too far away from the target object, thereby increasing the time cost and training cost of subsequent fine-tuning of the initial speech feature vector of the target object in the training stage facing the target object. Therefore, by removing the preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors, it is ensured that the fused speech feature vector can represent a better representation space of a speech feature suitable for any preset object, and will not be biased towards any preset object. Through this step, the time cost of subsequent fine-tuning of the initial speech feature vector of the target object can be reduced.

[0123] Specifically, the preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors is removed through the following steps:

[0124] Step one, it is judged whether there is a preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors.

[0125] Step two, if there is, for each preset speech feature vector, the number of other preset speech feature vectors with a difference degree greater than a preset difference degree from the preset speech feature vector is counted to obtain a difference degree corresponding to the preset speech feature vector.

[0126] Step three, after removing the preset speech feature vector with the largest difference degree, it is continuously judged whether there is a preset speech feature vector with a difference degree greater than a preset difference degree from other preset speech feature vectors until the difference degrees between the preset speech feature vectors are all less than the preset difference degree.

[0127] In the embodiments of the present application, the spatial distance between the preset speech feature vectors can be taken as the difference degree, and the preset difference degree can be a spatial distance threshold. The distance between the preset speech feature vectors can be a cosine distance, an Euclidean distance, a Manhattan distance, etc.

[0128] S103, input the target speech feature vector and the text to be synthesized into a speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

[0129] Referring to Figure 3As shown, a flowchart of synthesizing a target voice by a voice synthesis model provided by an embodiment of the present application is shown; the voice synthesis model comprises a text encoder, an attention model and a decoder. The process of synthesizing a target voice by the voice synthesis model is as follows:

[0130] Step one, input the text to be synthesized into the text encoder in the voice synthesis model to obtain a text vector to be synthesized.

[0131] In the embodiment of the present application, the text encoder can perform word segmentation and encoding on the text to be synthesized based on a BPE word segmentation technology or other encoding methods to obtain a text vector to be synthesized.

[0132] Step two, input the target voice feature vector into the attention model in the voice synthesis model, so that the attention model generates an auxiliary voice feature vector of an auxiliary voice feature dimension based on the text vector to be synthesized, and the auxiliary voice feature dimension and the target voice feature dimension constitute the total voice feature dimension; the attention model aligns the text vector to be synthesized with the target voice feature vector and the auxiliary voice feature vector.

[0133] In the embodiment of the present application, the target voice feature vector is a static voice feature dimension of the target object, the auxiliary voice feature dimension is a dynamic voice feature dimension of the target object, and the auxiliary voice feature vector includes feature values corresponding to each auxiliary voice feature dimension.

[0134] Here, (1) the static voice feature dimension is used to tell the voice synthesis model which voice to synthesize, determines the timbre recognition of the target object, and is used to lock the identity of the target object. The static voice feature dimension is input into the voice synthesis model once, telling the model that "this is the voice of target object A". The static voice feature dimension represents a global, time-invariant quantity, such as target object embedding, average fundamental frequency, formant distribution, global energy, etc. As long as the static voice feature dimension does not drift, the timbre, gender and age of the entire sentence will remain consistent; if it is distorted, "voice change" will occur - the text of A is spoken in the voice of B. (2) The dynamic voice feature dimension is a time variable used to describe "how to read this text to be synthesized". The dynamic voice feature dimension is injected into the voice in real time frame by frame / phoneme. The dynamic voice feature dimension represents a time-varying quantity, such as frame-level fundamental frequency curve, energy envelope, phoneme duration, emotion prosody label. The dynamic voice feature dimension determines the intonation fluctuation, emphasis position, pause length, emotional color; if it is missing or incorrect, even if the timbre is correct, it will sound like a "flat reading machine". The dynamic voice feature dimension is generated in real time by the attention or alignment module, so it is called the "auxiliary" dimension. The influence of the dynamic voice feature dimension on the listening: a, determines the intonation, emotion and clarity; b, if the dynamic dimension is distorted, the intonation will be flat, the emphasis will be missing, and it will sound mechanical, even if the timbre is correct, it will be "soulless".

[0135] Step three, input the corresponding relationship between the aligned to-be-synthesized text vector and the target speech feature vector and the auxiliary speech feature vector into a decoder in the speech synthesis model to obtain the target speech of the to-be-synthesized text.

[0136] Here, the speech synthesis model in the present application replaces the speaker encoder with a large number of parameters in the traditional TTS model by the target speech feature vector, and only retains the text encoder to process the to-be-synthesized text, thereby improving the speech synthesis efficiency.

[0137] The embodiment of the present application provides a speech synthesis method and device, the method comprising: fixing parameters of a speech synthesis model, taking higher similarity between real speech of a target object and speech of the target object synthesized by the speech synthesis model as an optimization target, updating and iterating feature values of an initial speech feature vector to obtain a target speech feature vector of the target object; and inputting the target speech feature vector and to-be-synthesized text into the speech synthesis model to synthesize target speech corresponding to the to-be-synthesized text. The present application introduces an initial speech feature vector with a smaller number of parameters, rather than training all parameters in the speech synthesis model for each target object, and uses the speech feature vector to represent the speech features of different speakers by adjusting the feature values of the initial speech feature vector, thereby synthesizing the speech of the target object. The scheme of the present application can reduce the training cost of copying different speaker timbres, shorten the synthesis time, and efficiently adapt to the speech synthesis task of different target objects.

[0138] Based on the same inventive concept, the embodiment of the present application also provides a speech synthesis device corresponding to the speech synthesis method. Since the principle of solving problems in the device of the embodiment of the present application is similar to the above-mentioned speech synthesis method of the embodiment of the present application, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0139] Referring to Figure 4 Fig. 1 shows a schematic diagram of a speech synthesis device provided by the embodiment of the present application, which comprises:

[0140] The acquisition module 401 is configured to acquire to-be-synthesized text corresponding to a target object.

[0141] The updating module 402 is configured to fix parameters of a speech synthesis model, take higher similarity between real speech of the target object and speech of the target object synthesized by the speech synthesis model as an optimization target, update and iterate feature values of an initial speech feature vector to obtain a target speech feature vector of the target object; and the text corresponding to the real speech is the same as the text corresponding to the synthesized speech.

[0142] The synthesis module 403 is configured to input the target speech feature vector and the text to be synthesized into the speech synthesis model, and synthesize target speech corresponding to the text to be synthesized.

[0143] The device can reduce the training cost and shorten the synthesis time by adjusting the feature values of the initial speech feature vector to synthesize the speech of the target object, and efficiently adapt to different speech synthesis tasks of different target objects.

[0144] As shown in Figure 5 The electronic device 500 provided by the embodiment of the present application includes a processor 501, a memory 502 and a bus. The memory 502 stores machine readable instructions executable by the processor 501. When the electronic device is running, the processor 501 and the memory 502 communicate through the bus. The processor 501 executes the machine readable instructions to perform the steps of the speech synthesis method described above.

[0145] Specifically, the memory 502 and the processor 501 can be general memory and processor, which are not limited here. When the processor 501 runs the computer program stored in the memory 502, the speech synthesis method described above can be executed.

[0146] Corresponding to the speech synthesis method described above, the embodiment of the present application further provides a computer readable storage medium, which stores a computer program. When the computer program is run by a processor, the steps of the speech synthesis method described above are executed.

[0147] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, which will not be described in detail in the present application. In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and actual implementation can have another division way, for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the displayed or discussed each other can be indirect coupling or communication connection through some communication interface, device or module, which can be electrical, mechanical or other forms.

[0148] The modules described as separate components may or may not be physically separate, and the components displayed as modules may or may not be physical units, i.e., may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiment.

[0149] In addition, each functional unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present alone, or two or more units can be integrated in one unit.

[0150] If the functions are realized in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application essentially or the part that contributes to the prior art or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the information processing method described in each embodiment of the present application. The foregoing storage medium includes: U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, and various program code storage media.

[0151] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A speech synthesis method, characterized in that, The method includes: Obtain the text to be synthesized corresponding to the target object; With the parameters of the fixed speech synthesis model as the optimization objective, the feature values ​​of the initial speech feature vector are updated and iterated to obtain the target speech feature vector of the target object, with the goal of achieving a higher similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model. The text corresponding to the real speech is the same as the text corresponding to the speech. The target speech feature vector and the text to be synthesized are input into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized; The initial speech feature vector is obtained through the following steps: feature fusion is performed on the preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector; the preset object includes the target object; the preset speech feature vector and the fused speech feature vector include the first feature value of each preset speech feature dimension; at least one target speech feature dimension is selected from all preset speech features according to the weight of each preset speech feature dimension; the weight is used to measure the degree to which the preset speech feature dimension can reflect the unique speech features of the target object; the first feature value corresponding to the target speech feature dimension is extracted from the fused speech feature vector to form the initial speech feature vector.

2. The speech synthesis method according to claim 1, characterized in that, The step of updating and iterating the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object includes: Obtain the first speech sample of the target object and the text sample corresponding to the first speech sample; For each update iteration of the feature values, the text sample and the latest initial speech feature vector are input into the speech synthesis model to obtain synthesized speech; The loss value of the speech synthesis model is calculated based on the first speech sample and the synthesized speech; the smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech. With the parameters of the speech synthesis model fixed, and with the minimum loss value of the speech synthesis model as the optimization objective, the feature values ​​of the latest initial speech feature vector are updated based on the loss value.

3. The speech synthesis method according to claim 1, characterized in that, The weights of each preset speech feature dimension for the target object are determined through the following steps: Extract the sample speech feature vector corresponding to the second speech sample of the target object; the sample speech feature vector includes the second feature values ​​of each preset speech feature dimension; Speech feature decomposition is performed on the speech feature vectors of each sample to obtain the second feature value of each sample speech feature vector under each preset speech feature dimension; For each preset speech feature dimension, the weight of the preset speech feature dimension to the target object is calculated based on the second feature value of the preset speech feature dimension in each sample speech feature vector.

4. The speech synthesis method according to claim 3, characterized in that, The step of calculating the weight of the preset speech feature dimension with respect to the target object based on the second feature value of the preset speech feature dimension in each sample speech feature vector includes: Calculate the variance of all second feature values ​​of the preset speech feature dimension to obtain the weight of the preset speech feature dimension with respect to the target object.

5. The speech synthesis method according to claim 1, characterized in that, The step of selecting the target speech feature dimension from all preset speech features based on the weights of each preset speech feature dimension for the target object includes: The weights of the target object are sorted according to all preset speech feature dimensions; The number of preset speech feature dimensions with the highest weight are determined as the target speech feature dimensions.

6. The speech synthesis method according to claim 1, characterized in that, Before performing feature fusion on the preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector, the method further includes: Remove preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference, and obtain the final preset speech feature vector.

7. The speech synthesis method according to claim 6, characterized in that, The removal of preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference includes: Determine whether there exists a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference. If they exist, for each preset speech feature vector, count the number of other preset speech feature vectors whose difference degree is greater than the preset difference degree, and obtain the degree of difference corresponding to the preset speech feature vector. After removing the preset speech feature vector with the greatest difference, it continues to determine whether there are preset speech feature vectors with a difference greater than the preset difference between them and other preset speech feature vectors, until the difference between the preset speech feature vectors is less than the preset difference.

8. The speech synthesis method according to claim 1, characterized in that, The step of inputting the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized includes: The text to be synthesized is input into the text encoder in the speech synthesis model to obtain the text vector to be synthesized; The target speech feature vector is input into the attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector corresponding to the auxiliary speech feature dimension based on the text vector to be synthesized. The auxiliary speech feature dimension and the target speech feature dimension constitute the total speech feature dimension. The attention model aligns the text vector to be synthesized with the target speech feature vector and the auxiliary speech feature vector. The correspondence between the aligned text vector to be synthesized and the target speech feature vector and the auxiliary speech feature vector is input into the decoder in the speech synthesis model to obtain the target speech of the text to be synthesized.

9. A speech synthesis device, characterized in that, The device includes: The acquisition module is used to acquire the text to be synthesized corresponding to the target object; The update module is used to fix the parameters of the speech synthesis model, with the optimization goal of increasing the similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model. The feature values ​​of the initial speech feature vector are updated and iterated to obtain the target speech feature vector of the target object; the text corresponding to the real speech is the same as the text corresponding to the speech. The synthesis module is used to input the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized; The updating module is further configured to: perform feature fusion on preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector; the preset object includes the target object; the preset speech feature vector and the fused speech feature vector include first feature values ​​of each preset speech feature dimension; select at least one target speech feature dimension from all preset speech features according to the weights of each preset speech feature dimension; the weights are used to measure the degree to which the preset speech feature dimension can reflect the unique speech features of the target object; extract the first feature values ​​corresponding to the target speech feature dimension from the fused speech feature vector to form an initial speech feature vector.

Images