Speech synthesis method and device

Abstract

The invention relates to the technical field of speech synthesis, in particular to a speech synthesis method and device, and the method comprises the steps: fixing the parameters of a speech synthesis model, and taking the higher similarity between the real speech of a target object and the speech of the target object synthesized by the speech synthesis model as an optimization target; updating and iterating the feature value of the initial voice feature vector to obtain a target voice feature vector of the target object; and inputting the target speech feature vector and the to-be-synthesized text into a speech synthesis model, and synthesizing a target speech corresponding to the to-be-synthesized text. The speech of the target object is synthesized by adjusting the feature value of the initial speech feature vector, so that the training cost can be reduced, the synthesis time consumption can be shortened, and speech synthesis tasks of different target objects can be efficiently adapted.

Description

Technical Field

[0001] The present invention relates to the field of speech synthesis technology, and in particular to a speech synthesis method and device. Background Art

[0002] Text-to-speech (TTS) is a technology that converts text into speech. In existing technologies, when a TTS model is required to synthesize the speech of a specific subject, it first extracts speech feature vectors from the subject's speech sample. Then, based on these speech feature vectors, the parameters in the TTS model are fine-tuned.

[0003] However, to ensure the accuracy and naturalness of synthesized speech from different speakers, the TTS model requires repeated training of numerous parameters. A set of parameters in a model can only be adapted to a single target speaker. This results in high training costs and time consumption, making it difficult to efficiently adapt speech synthesis tasks for different speakers. Summary of the Invention

[0004] In view of this, the purpose of this application is to provide a speech synthesis method and device that can reduce training costs, shorten training time, and efficiently adapt to speech synthesis tasks for different speakers.

[0005] In a first aspect, an embodiment of the present application provides a speech synthesis method, the method comprising: Get the text to be synthesized corresponding to the target object; The parameters of the speech synthesis model are fixed, and the similarity between the real speech of the target subject and the speech of the target subject synthesized by the speech synthesis model is optimized to be higher. The feature values ​​of the initial speech feature vector are updated iteratively to obtain the target speech feature vector of the target subject; the text corresponding to the real speech is the same as the text corresponding to the speech; The target speech feature vector and the text to be synthesized are input into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

[0006] In a possible implementation, the iterative updating of the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object includes: Acquire a first voice sample of the target object and a text sample corresponding to the first voice sample; For each update iteration of the feature value, the text sample and the latest initial speech feature vector are input into the speech synthesis model to obtain synthesized speech; Calculating a loss value of the speech synthesis model based on the first speech sample and the synthesized speech; the smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech; The parameters of the speech synthesis model are fixed, and the loss value of the speech synthesis model is minimized as an optimization goal. The feature value of the latest initial speech feature vector is updated according to the loss value.

[0007] In a possible implementation, the initial speech feature vector is obtained by the following steps: Performing feature fusion on preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector; the preset object includes the target object; the preset speech feature vector and the fused speech feature vector include first feature values ​​of each preset speech feature dimension; At least one target speech feature dimension is selected from all preset speech features according to the weights of the preset speech feature dimensions; the weights are used to measure the extent to which the preset speech feature dimensions can reflect the unique speech features of the target subject; The first feature value corresponding to the target speech feature dimension is extracted from the fused speech feature vector to form an initial speech feature vector.

[0008] In a possible implementation, the weight of each preset speech feature dimension to the target object is determined by the following steps: Extracting a sample speech feature vector corresponding to a second speech sample of the target object; the sample speech feature vector includes second feature values ​​of each preset speech feature dimension; Performing speech feature decomposition on each sample speech feature vector to obtain a second feature value of each sample speech feature vector in each preset speech feature dimension; For each preset speech feature dimension, a weight of the preset speech feature dimension to the target object is calculated according to a second feature value of the preset speech feature dimension in each sample speech feature vector.

[0009] In a possible implementation, calculating the weight of the preset speech feature dimension for the target object according to the second feature value of the preset speech feature dimension in each sample speech feature vector includes: The variance of all second feature values ​​of the preset speech feature dimension is calculated to obtain the weight of the preset speech feature dimension to the target object.

[0010] In a possible implementation, the step of selecting a target speech feature dimension from all preset speech features according to the weight of each preset speech feature dimension on the target object includes: Sort all preset speech feature dimensions by weighting the target object; A preset number of preset speech feature dimensions with the largest weights are determined as target speech feature dimensions.

[0011] In a possible implementation, before performing feature fusion on the preset speech feature vectors of the preset speech sample of at least one preset object to obtain the fused speech feature vector, the method further includes: Preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference are removed to obtain a final preset speech feature vector.

[0012] In a possible implementation, removing a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference includes: Determining whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference; If so, for each preset speech feature vector, count the number of other preset speech feature vectors whose difference from the preset speech feature vector is greater than the preset difference, to obtain the difference degree corresponding to the preset speech feature vector; After removing the preset speech feature vector with the largest difference, continue to determine whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than the preset difference, until the differences between the preset speech feature vectors are all less than the preset difference.

[0013] In a possible implementation, inputting the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized includes: Inputting the text to be synthesized into the text encoder in the speech synthesis model to obtain a text vector to be synthesized; Inputting the target speech feature vector into the attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector of an auxiliary speech feature dimension based on the text vector to be synthesized, wherein the auxiliary speech feature dimension and the target speech feature dimension constitute all speech feature dimensions; the attention model aligns the text vector to be synthesized with the target speech feature vector and the auxiliary speech feature vector; The correspondence between the aligned text vector to be synthesized, the target speech feature vector and the auxiliary speech feature vector is input into the decoder in the speech synthesis model to obtain the target speech of the text to be synthesized.

[0014] In a second aspect, an embodiment of the present application further provides a speech synthesis device, the device comprising: An acquisition module is used to obtain the text to be synthesized corresponding to the target object; An updating module, configured to fix parameters of the speech synthesis model, iteratively update the feature values ​​of the initial speech feature vector with the optimization goal of achieving a higher similarity between the real speech of the target subject and the speech of the target subject synthesized by the speech synthesis model, and obtain a target speech feature vector of the target subject; the text corresponding to the real speech is the same as the text corresponding to the speech; The synthesis module is used to input the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

[0015] In one possible implementation, the update module is specifically used to obtain a first speech sample of the target object and a text sample corresponding to the first speech sample; for each update iteration of the feature value, the text sample and the latest initial speech feature vector are input into the speech synthesis model to obtain synthesized speech; the loss value of the speech synthesis model is calculated based on the first speech sample and the synthesized speech; the smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech; the parameters of the speech synthesis model are fixed, and the minimization of the loss value of the speech synthesis model is taken as the optimization goal, and the feature value of the latest initial speech feature vector is updated according to the loss value.

[0016] In a possible implementation, the update module is further configured to: Performing feature fusion on preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector; the preset object includes the target object; the preset speech feature vector and the fused speech feature vector include first feature values ​​of each preset speech feature dimension; At least one target speech feature dimension is selected from all preset speech features according to the weights of the preset speech feature dimensions; the weights are used to measure the extent to which the preset speech feature dimensions can reflect the unique speech features of the target subject; The first feature value corresponding to the target speech feature dimension is extracted from the fused speech feature vector to form an initial speech feature vector.

[0017] In a possible implementation, the update module is further configured to: Extracting a sample speech feature vector corresponding to a second speech sample of the target object; the sample speech feature vector includes second feature values ​​of each preset speech feature dimension; Performing speech feature decomposition on each sample speech feature vector to obtain a second feature value of each sample speech feature vector in each preset speech feature dimension; For each preset speech feature dimension, a weight of the preset speech feature dimension to the target object is calculated according to a second feature value of the preset speech feature dimension in each sample speech feature vector.

[0018] In a possible implementation, the update module is further configured to: The variance of all second feature values ​​of the preset speech feature dimension is calculated to obtain the weight of the preset speech feature dimension to the target object.

[0019] In a possible implementation, the update module is further configured to: Sort all preset speech feature dimensions by weighting the target object; A preset number of preset speech feature dimensions with the largest weights are determined as target speech feature dimensions.

[0020] In a possible implementation, before performing feature fusion on the preset speech feature vectors of the preset speech sample of at least one preset object to obtain the fused speech feature vector, the updating module is further configured to: Preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference are removed to obtain a final preset speech feature vector.

[0021] In a possible implementation, the update module is further configured to: Determining whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference; If so, for each preset speech feature vector, count the number of other preset speech feature vectors whose difference from the preset speech feature vector is greater than the preset difference, to obtain the difference degree corresponding to the preset speech feature vector; After removing the preset speech feature vector with the largest difference, continue to determine whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than the preset difference, until the differences between the preset speech feature vectors are all less than the preset difference.

[0022] In one possible embodiment, the synthesis module is specifically used to input the text to be synthesized into the text encoder in the speech synthesis model to obtain a text vector to be synthesized; input the target speech feature vector into the attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector of an auxiliary speech feature dimension based on the text vector to be synthesized, and the auxiliary speech feature dimension and the target speech feature dimension constitute all speech feature dimensions; the attention model aligns the text vector to be synthesized with the target speech feature vector and the auxiliary speech feature vector; inputs the correspondence between the text vector to be synthesized obtained after alignment and the target speech feature vector and the auxiliary speech feature vector into the decoder in the speech synthesis model to obtain the target speech of the text to be synthesized.

[0023] The embodiment of the present application provides a speech synthesis method and device, which includes: fixing the parameters of the speech synthesis model, taking the similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model as the optimization goal, updating and iterating the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object; inputting the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized. The present application introduces an initial speech feature vector with a smaller number of parameters, rather than training all the parameters in the speech synthesis model once for each target object. By adjusting the feature values ​​of the initial speech feature vector, the speech feature vector can be used to represent the speech characteristics of different speakers, and then the speech of the target object can be synthesized. The solution of the present application can reduce the training cost of replicating the timbre of different speakers, shorten the synthesis time, and efficiently adapt to the speech synthesis tasks of different target objects. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without creative work.

[0025] Figure 1 A flowchart of a speech synthesis method provided by an embodiment of the present application is shown; Figure 2 A flowchart of iteratively updating the feature values ​​of the initial speech feature vector provided by an embodiment of the present application is shown; Figure 3 A schematic diagram of a process for synthesizing a target speech using a speech synthesis model provided in an embodiment of the present application is shown; Figure 4A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application is shown; Figure 5 A schematic structural diagram of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of illustration and description and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn to scale. The flowcharts used in this application illustrate the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can be implemented out of sequence, and steps without logical context can be reversed or implemented simultaneously. In addition, those skilled in the art, under the guidance of the contents of this application, can add one or more other operations to the flowchart, or remove one or more operations from the flowchart.

[0027] In addition, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present application.

[0028] To enable those skilled in the art to use the contents of this application, the following embodiments are provided in conjunction with a specific application scenario, the "field of speech synthesis technology." Those skilled in the art will appreciate that the general principles defined herein may be applied to other embodiments and application scenarios without departing from the spirit and scope of this application. Although this application is primarily described in the "field of speech synthesis technology," it should be understood that this is merely an exemplary embodiment and may also be applied to, for example, the field of song synthesis technology.

[0029] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the existence of the features declared thereafter, but does not exclude the addition of other features.

[0030] The following is a detailed description of a speech synthesis method provided in an embodiment of the present application.

[0031] Reference Figure 1FIG. 1 is a flow chart of a speech synthesis method provided in an embodiment of the present application. The exemplary steps of the embodiment of the present application are described below: S101: Obtain the text to be synthesized corresponding to the target object.

[0032] In the embodiment of the present application, the target object can be any object that can make a sound, such as a person, a bird, a tiger, etc. The text to be synthesized refers to the text that needs to be synthesized into speech, such as "hello".

[0033] S102. Fix the parameters of the speech synthesis model, take the similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model as the optimization goal, update the feature values ​​of the initial speech feature vector iteratively, and obtain the target speech feature vector of the target object.

[0034] In an embodiment of the present application, the target subject's real speech is a natural speech sample actually uttered by the target subject, and the text corresponding to the real speech is identical to the text corresponding to the target subject's speech synthesized by the speech synthesis model. The initial speech feature vector includes a feature value of at least one preset speech feature dimension. The preset speech feature dimension corresponding to the initial speech feature vector is a speech feature dimension that can be used to distinguish the target speaker. For example, if the initial speech feature vector includes a feature value of timbre, a feature value of rhythm, and a feature value of speech rate, then timbre, rhythm, and speech rate are speech feature dimensions that can be used to distinguish the target speaker. The speech synthesis model uses the initial speech feature vector or the target speech feature vector to synthesize the target subject's speech. The higher the similarity between the target subject's real speech and the target subject's speech synthesized by the speech synthesis model, the more capable the initial speech feature vector or the target speech feature vector is of distinguishing the target speaker. The target speech feature vector is a speech feature vector that can be used to distinguish the target speaker.

[0035] The preset speech feature dimensions corresponding to the initial speech feature vector are determined based on the degree to which each preset speech feature dimension can reflect the unique speech characteristics of the target object. For example, in a preferred embodiment, the preset speech feature dimensions include speech-related representations such as timbre, prosody, speaking rate, intonation, and pitch (frequency). All preset speech feature dimensions are sorted according to the degree to which they can reflect the unique speech characteristics of target object X. The order is timbre > intonation > speaking rate > prosody > pitch. The first three are taken as the preset speech feature dimensions corresponding to the initial speech feature vector. Therefore, the preset speech feature dimensions corresponding to target object X include timbre, intonation, and speaking rate.

[0036] In the prior art, the speech feature vector corresponding to the target object is usually extracted from a large number of speech samples of the target object, and then a large number of parameters in the speech synthesis model are fine-tuned based on these speech feature vectors. In contrast, the number of parameters in the TTS model is much larger than the number of parameters in the initial speech feature vector. Therefore, compared with the traditional method of synthesizing the speech of the target object by fine-tuning the parameters in the TTS model, the method of synthesizing the speech of the target object by adjusting the feature values ​​of the initial speech feature vector in the present application can reduce training costs, shorten synthesis time, and efficiently adapt to speech synthesis tasks for different target objects.

[0037] Reference Figure 2 As shown in FIG, it is a flowchart for iterating the update of the feature value of the initial speech feature vector provided in an embodiment of the present application, and the specific steps are as follows: S201: Obtain a first voice sample of a target object and a text sample corresponding to the first voice sample.

[0038] In the embodiment of the present application, the first voice sample is a natural voice sample actually uttered by the target object. For example, if the first voice sample is the natural voice of "eat snacks" actually uttered by the target object, the text sample corresponding to the first voice sample is "eat snacks".

[0039] S202: For each update iteration of the feature value, the text sample and the latest initial speech feature vector are input into the speech synthesis model to obtain synthesized speech.

[0040] S203. Calculate a loss value of the speech synthesis model according to the first speech sample and the synthesized speech.

[0041] In this embodiment of the present application, the loss value of the speech synthesis model is used to quantify the difference between the first speech sample and the synthesized speech. The smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech. This indicates that the latest initial speech feature vector is more capable of distinguishing the target speaker and the better the performance when synthesizing the target subject's speech.

[0042] Among them, the loss value of the speech synthesis model can be calculated through loss functions such as L1 Loss, L2 Loss, and STFT Loss.

[0043] S204 , fixing the parameters of the speech synthesis model, taking the minimum loss value of the speech synthesis model as the optimization goal, and updating the feature value of the latest initial speech feature vector according to the loss value.

[0044] Here, the "initial speech feature vector" is a fixed-size container containing feature values ​​of at least one preset speech feature dimension. For different target objects, the preset speech feature dimensions and / or feature values ​​in the initial speech feature vector container are fine-tuned, and the speech features of different target objects are characterized by the feature values ​​of different preset speech feature dimensions. The "initial speech feature vector" replaces the audio encoder and the sound feature extraction model in the traditional TTS model. The number of parameters of the "initial speech feature vector" is significantly lower than the number of parameters of the audio encoder and the sound feature extraction model. This application does not require the fine-tuning of the attention model and decoder parameters in the traditional TTS model. It only needs to fine-tune the "initial speech feature vector" for different target objects. The number of parameters that need to be fine-tuned is greatly reduced compared to the TTS model, which significantly reduces the training overhead.

[0045] It should be noted that this application still needs to fine-tune different "initial speech feature vectors" for different target objects. Fine-tuning the first speech sample of an "initial speech feature vector" requires a relatively stable timbre performance corresponding to a fixed target object. Therefore, it is necessary to ensure that the similarity between each first speech sample meets the preset similarity requirements.

[0046] In addition, the present application obtains the initial speech feature vector through the following steps: Step 1: performing feature fusion on preset speech feature vectors of a preset speech sample of at least one preset object to obtain a fused speech feature vector.

[0047] In the embodiment of the present application, the preset object includes a target object. A preset object can have preset voice feature vectors of one or more preset voice samples. The preset voice sample is a natural voice sample actually uttered by the preset object, such as the second voice sample is the natural voice of "Have you eaten?" actually uttered by the target object. The preset voice feature vector is extracted by inputting the preset voice sample into the voice feature vector extraction model. The preset voice feature vector and the fused voice feature vector include the first feature value of each preset voice feature dimension. All preset voice feature vectors have the same dimension to facilitate subsequent processing.

[0048] As an example, the speech vector extraction model may adopt a model such as HuBERT, Wav2Vec series, or x-vectors.

[0049] For example, the preset speech feature dimensions include timbre, rhythm, speaking rate, intonation, and pitch (frequency), and the preset speech feature vector includes the first feature value of timbre, the first feature value of rhythm, the first feature value of speaking rate, the first feature value of intonation, and the first feature value of pitch.

[0050] Here, through feature fusion, it can be ensured that the fused speech feature vector can represent a better representation space of the speech features adapted to any preset object, and will not be biased towards any preset object. This step can reduce the time cost of subsequent fine-tuning of the initial speech feature vector of the target object.

[0051] The preset voice samples are obtained after preprocessing. For example, if some preset voice samples are too long, the preset voice samples can be segmented according to the preset upper limit of the voice sample length. The preset voice samples can also be preprocessed by audio noise reduction, audio enhancement, etc.

[0052] Specifically, the fusion may be performed by adopting an average feature method, specifically, averaging a plurality of preset speech feature vectors corresponding to a plurality of preset speech samples of at least one preset object to obtain a fused speech feature vector.

[0053] For example, assuming that the preset speech feature vector dimensions of the preset speech sample a and the preset speech sample b are aligned, the preset speech feature vector of the preset speech sample a of the preset object A is [3, 8, 7], and the preset speech feature vector of the preset speech sample b of the preset object B is [7, 4, 5], then the preset speech feature vector of the preset speech sample a of the preset object A and the preset speech feature vector of the preset speech sample b of the preset object B are averaged to obtain the fused speech feature vector [5, 6, 6].

[0054] Step 2: According to the weights of the preset speech feature dimensions, at least one target speech feature dimension is selected from all the preset speech features.

[0055] In the embodiment of the present application, the weight of each preset speech feature dimension is used to measure the extent to which the preset speech feature dimension can reflect the unique speech feature of the target object.

[0056] For example, the preset speech feature dimensions include common speech-related representations such as timbre, prosody, speaking rate, intonation, pitch (frequency), and loudness. Among them, timbre can include features such as fundamental frequency and formant. In addition, it can also include less common features such as harmonic-to-noise ratio, speaking style, and accent. Taking common representations as an example, assuming that the representation weight of timbre is 0.6, the representation weight of prosody is 0.9, the representation weight of speaking rate is 0.2, the representation weight of intonation is 0.1, and the representation weight of pitch is 0.7, the preset speech feature dimension with a representation weight greater than 0.6 is taken as the target speech feature dimension. The target speech feature dimension then includes prosody and pitch. While highlighting the speech characteristics of the target object, it can further reduce the number of parameters in the speech feature vector, thereby reducing training overhead.

[0057] Specifically, according to the weight of each preset voice feature dimension on the target object, the target voice feature dimension is screened out from all preset voice features, including: i. Sort all preset speech feature dimensions by weighting the target object; ii. Determine a preset number of preset speech feature dimensions with the largest weights as target speech feature dimensions.

[0058] For example, according to the weight of each preset speech feature dimension, pitch > timbre > intonation > speaking speed > rhythm, the first three are taken as the preset speech feature dimensions corresponding to the initial speech feature vector. Therefore, the preset speech feature dimensions corresponding to the target object X include timbre, intonation and speaking speed.

[0059] In addition, the weight of each preset speech feature dimension to the target object is determined by the following steps: i. Extracting a sample speech feature vector corresponding to the second speech sample of the target object.

[0060] In the embodiments of the present application, the second speech sample is a natural speech sample actually uttered by the target subject, such as the natural speech of "good morning" actually uttered by the target subject. The sample speech feature vector is extracted by inputting the second speech sample into the speech feature vector extraction model, and includes the second feature values ​​of each preset speech feature dimension.

[0061] For example, the preset speech feature dimensions include timbre, rhythm, speaking rate, intonation, and pitch (frequency), then the parameters of the sample speech feature vector can represent the second characteristic value of the timbre, the second characteristic value of the rhythm, the second characteristic value of the speaking rate, the second characteristic value of the intonation, and the second characteristic value of the pitch of the target object.

[0062] ii. Perform speech feature decomposition on each sample speech feature vector to obtain a second feature value of each sample speech feature vector in each preset speech feature dimension.

[0063] In the embodiment of the present application, the high-dimensional sample speech feature vector is mapped to an interpretable and controllable subspace of a preset speech feature dimension through feature decomposition, that is, mapped to the second feature value under each preset speech feature dimension.

[0064] In this example, the preset speech feature dimensions include timbre, rhythm, speaking speed, intonation, and pitch (frequency). The sample speech feature vector R is [3, 7, 0, 2, 7]. The sample speech feature vector R is decomposed into speech features to obtain the second feature value of the sample speech feature vector R under timbre as 3, the second feature value of the sample speech feature vector R under rhythm as 7, the second feature value of the sample speech feature vector R under speaking speed as 0, the second feature value of the sample speech feature vector R under intonation as 2, and the second feature value of the sample speech feature vector R under pitch as 7.

[0065] It should be understood that the above examples are only for ease of understanding. In fact, the feature vectors in this application express the sound characteristics of the speaking object through a lower-dimensional dense representation. The speech feature vector is usually a latent vector or embedded vector, which is an abstract representation encoded by the speech vector extraction model after learning a large number of samples. The numerical values ​​of each dimension have no intuitive physical meaning.

[0066] iii. For each preset speech feature dimension, calculate the weight of the preset speech feature dimension to the target object according to the second feature value of the preset speech feature dimension in each sample speech feature vector.

[0067] In an embodiment of the present application, the variance of all second feature values ​​of the preset speech feature dimension is calculated to obtain the weight of the preset speech feature dimension to the target object.

[0068] Here, the role of variance in synthesizing the speech of the target object by the preset synthesis model includes: (1) Capturing the key speech features of the target object: A preset speech feature dimension with a large variance indicates that the sample speech feature vector changes more significantly in this direction, and can capture the main variability in the sample speech feature vector. In speech synthesis, the speech features of the target object (such as timbre, pitch, resonance peak, etc.) are the key to distinguishing different objects. A preset speech feature dimension with a large variance can better capture these key features, thereby retaining the unique "personality" of the target object when synthesizing speech. (2) Improving the naturalness of the synthesized speech: The naturalness of the synthesized speech is one of the important indicators for measuring the performance of the speech synthesis system. Natural speech is more easily accepted by users, especially in scenarios that require long-term interaction (such as smart assistants, audiobooks, etc.). A preset speech feature dimension with a large variance can retain more features related to the natural speech of the target object, making the synthesized speech closer to the speech of the target object in terms of timbre, rhythm and intonation, thereby improving naturalness. (3) Enhance speaker recognizability: By retaining the preset speech feature dimensions with large variance, the synthesized speech can better simulate the timbre and speech characteristics of the target subject, thereby enhancing the recognizability of the target subject. In summary, in the field of speech synthesis, the preset speech feature dimensions with large variance have significant advantages for the speech synthesis of the target subject. They can better capture the key speech features of the target subject and improve the naturalness and recognizability of the synthesized target subject's speech.

[0069] Step 3: Extract the first feature value corresponding to the target speech feature dimension from the fused speech feature vector to form an initial speech feature vector.

[0070] For example, the preset speech feature dimensions include timbre, rhythm, speaking rate, intonation, and pitch (frequency). The fused speech feature vector is [3, 7, 9, 2, 8]. The target speech feature dimensions are rhythm and speaking rate, so the initial speech feature vector is [7, 9].

[0071] In the embodiments of the present application, different first difference parameters can be set for speech samples of different target subjects. The setting of the first difference parameter is related to the uniqueness (or scarcity) of the preset speech feature vector of the preset speech sample of the preset subject, the audio quality of the preset speech sample, etc. For example, for subjects with more distinctive accents, or dialects or languages ​​with very few speakers, the preset initialization difference parameter can be a larger value, while for common languages, the preset initialization difference can be a smaller value. If the audio quality is higher, such as for subjects with higher sampling rates, sampling depths, or bit rates, the preset initialization difference parameter can be smaller. In a preferred embodiment, for a target object, a method for determining a first difference parameter may include: selecting a speech feature vector corresponding to a speech sample with the highest audio quality as a reference vector, and using the audio quality parameter as the horizontal axis. For example, a preset speech feature vector corresponding to a speech sample with the highest sampling rate is selected as a reference vector, then a scatter plot is drawn with the sampling rate parameter as the horizontal axis, and the spatial distance between the preset speech feature vector extracted from other speech samples and the reference vector as the vertical axis, and based on the scatter plot, a threshold position is determined. For example, if the spatial distances corresponding to 98% of the speech samples are below 0.1, then 0.1 is used as the 98th percentile point, which is the first difference parameter of the target object. When training for the same target object, only speech samples that meet the first difference parameter are used for training. While highlighting the speech characteristics of the target object, the training overhead for each target object is further reduced.

[0072] It should be understood that when performing speech synthesis using the solution of the present application, for different target objects, it is still necessary to train the initial speech feature vector based on the speech samples of each target object to obtain the target speech feature vector corresponding to each target object.

[0073] Furthermore, before performing feature fusion on the preset speech feature vectors of the preset speech samples of at least one preset object to obtain the fused speech feature vector, the method further includes: removing the preset speech feature vectors whose difference from other preset speech feature vectors is greater than the preset difference, to obtain the final preset speech feature vector. If the difference between the preset speech feature vectors is large, it may cause the fused speech feature vector to be biased towards some preset objects, thereby causing the initial speech feature vector to be too far away from the target object, increasing the time cost and training cost of the subsequent fine-tuning of the initial speech feature vector of the target object in the training phase for the target object. Therefore, the present application ensures that the fused speech feature vector can represent a better representation space that is adapted to the speech features of any preset object by removing the preset speech feature vectors whose difference from other preset speech feature vectors is greater than the preset difference, and will not be biased towards any preset object. This step can reduce the time cost of the subsequent fine-tuning of the initial speech feature vector of the target object.

[0074] Specifically, the following steps are performed to remove the preset speech feature vectors whose difference from other preset speech feature vectors is greater than the preset difference: Step 1: Determine whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference.

[0075] Step 2: If it exists, for each preset speech feature vector, count the number of other preset speech feature vectors whose difference from the preset speech feature vector is greater than the preset difference, and obtain the difference degree corresponding to the preset speech feature vector.

[0076] Step 3: After removing the preset speech feature vector with the largest difference, continue to determine whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than the preset difference, until the differences between the preset speech feature vectors are all less than the preset difference.

[0077] In the embodiment of the present application, the spatial distance between the preset speech feature vectors can be used as the difference, and the preset difference can be a spatial distance threshold. The distance between the preset speech feature vectors can be cosine distance, Euclidean distance, Manhattan distance, etc.

[0078] S103: Input the target speech feature vector and the text to be synthesized into a speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

[0079] Reference Figure 3 The figure shows a flow chart of synthesizing a target speech using a speech synthesis model provided in an embodiment of the present application; the speech synthesis model includes a text encoder, an attention model, and a decoder. The process of synthesizing the target speech using the speech synthesis model is as follows: Step 1: Input the text to be synthesized into the text encoder in the speech synthesis model to obtain the text vector to be synthesized.

[0080] In the implementation manner of the present application, the text encoder can perform word segmentation encoding on the text to be synthesized based on encoding methods such as BPE word segmentation technology to obtain a vector of the text to be synthesized.

[0081] Step 2: Input the target speech feature vector into the attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector of the auxiliary speech feature dimension based on the text vector to be synthesized, and the auxiliary speech feature dimension and the target speech feature dimension constitute all the speech feature dimensions; the attention model aligns the text vector to be synthesized with the target speech feature vector and the auxiliary speech feature vector.

[0082] In the embodiment of the present application, the target speech feature vector is the static speech feature dimension of the target object, the auxiliary speech feature dimension is the dynamic speech feature dimension of the target object, and the auxiliary speech feature vector includes feature values ​​corresponding to each auxiliary speech feature dimension.

[0083] Here, (1) the static speech feature dimension is used to tell the speech synthesis model whose voice is being synthesized, determine the timbre recognition of the target object, and lock the identity of the target object. The speech features of the static speech feature dimension are input into the speech synthesis model at one time, telling the model "this is the voice of target object A". The static speech feature dimension represents global, time-invariant quantities, such as target object embedding, average fundamental frequency, formant distribution, global energy, etc. As long as the static speech feature dimension does not drift, the timbre, gender, and age of the entire sentence will remain consistent; if it is distorted, "voice change" will occur - A's text is pronounced by B's voice. (2) The dynamic speech feature dimension is a time variable used to describe "how to read this text to be synthesized". The speech features of the dynamic speech feature dimension are injected into the speech in real time, frame by frame / phoneme by phoneme. The dynamic speech feature dimension represents time-varying quantities, such as frame-level fundamental frequency curve, energy envelope, phoneme duration, and emotional prosody markers. The dynamic speech feature dimension determines the rise and fall of intonation, stress placement, pause length, and emotional tone. If it is missing or incorrect, even if the timbre is correct, the voice will sound like a "flat machine." This dynamic speech feature dimension is generated in real time by the attention or alignment module, hence its term "auxiliary" dimension. Its impact on listening: a) It determines intonation, emotion, and clarity. b) If the dynamic dimension is distorted, the voice will sound flat, with missing stress and a robotic feel. Even if the timbre is correct, the voice will sound "soulless."

[0084] Step 3: Input the correspondence between the aligned text vector to be synthesized, the target speech feature vector, and the auxiliary speech feature vector into the decoder in the speech synthesis model to obtain the target speech of the text to be synthesized.

[0085] Here, the speech synthesis model in this application replaces the speaker encoder with a large number of parameters in the traditional TTS model with the target speech feature vector, and only retains the text encoder to process the text to be synthesized, thereby improving the speech synthesis efficiency.

[0086] The embodiment of the present application provides a speech synthesis method and device, which includes: fixing the parameters of the speech synthesis model, taking the similarity between the real speech of the target object and the speech of the target object synthesized by the speech synthesis model as the optimization goal, updating and iterating the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object; inputting the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized. The present application introduces an initial speech feature vector with a smaller number of parameters, rather than training all the parameters in the speech synthesis model once for each target object. By adjusting the feature values ​​of the initial speech feature vector, the speech feature vector can be used to represent the speech characteristics of different speakers, and then the speech of the target object can be synthesized. The solution of the present application can reduce the training cost of replicating the timbre of different speakers, shorten the synthesis time, and efficiently adapt to the speech synthesis tasks of different target objects.

[0087] Based on the same inventive concept, a speech synthesis device corresponding to the speech synthesis method is also provided in the embodiment of the present application. Since the principle of solving the problem by the device in the embodiment of the present application is similar to the above-mentioned speech synthesis method in the embodiment of the present application, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0088] Reference Figure 4 FIG. 1 is a schematic diagram of a speech synthesis device provided in an embodiment of the present application, wherein the speech synthesis device includes: An acquisition module 401 is used to acquire the text to be synthesized corresponding to the target object; An updating module 402 is configured to fix parameters of the speech synthesis model, iteratively update the feature values ​​of the initial speech feature vector with the goal of optimizing the similarity between the real speech of the target subject and the speech of the target subject synthesized by the speech synthesis model, and obtain a target speech feature vector of the target subject; the text corresponding to the real speech is identical to the text corresponding to the synthesized speech; The synthesis module 403 is configured to input the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

[0089] This device synthesizes the target object's speech by adjusting the feature values ​​of the initial speech feature vector, which can reduce training costs, shorten synthesis time, and efficiently adapt to speech synthesis tasks for different target objects.

[0090] like Figure 5 As shown, an electronic device 500 provided in an embodiment of the present application includes: a processor 501, a memory 502 and a bus, wherein the memory 502 stores machine-readable instructions executable by the processor 501. When the electronic device is running, the processor 501 communicates with the memory 502 through the bus, and the processor 501 executes the machine-readable instructions to perform the steps of the above-mentioned speech synthesis method.

[0091] Specifically, the memory 502 and the processor 501 can be general-purpose memories and processors, which are not specifically limited here. When the processor 501 runs the computer program stored in the memory 502, the speech synthesis method can be executed.

[0092] Corresponding to the above-mentioned speech synthesis method, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned speech synthesis method are executed.

[0093] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and device described above can refer to the corresponding process in the method embodiment, and will not be repeated in this application. In the several embodiments provided in this application, it should be understood that the disclosed system, device and method can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation. For example, multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or modules, which can be electrical, mechanical or other forms.

[0094] The modules described as separate components may or may not be physically separate, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed across multiple network elements. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0096] If the functions are implemented as software functional units and sold or used as independent products, they can be stored in a processor-executable, non-volatile, computer-readable storage medium. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or the portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the information processing method described in each embodiment of this application. The aforementioned storage medium includes various media that can store program code, such as USB flash drives, mobile hard drives, ROM, RAM, magnetic disks, or optical disks.

[0097] The above are only specific embodiments of the present application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in this application should be included in the scope of protection of this application. Therefore, the scope of protection of this application should be based on the scope of protection of the claims.

Claims

1. A speech synthesis method, characterized in that: The method comprises: Get the text to be synthesized corresponding to the target object; The parameters of the speech synthesis model are fixed, and the similarity between the real speech of the target subject and the speech of the target subject synthesized by the speech synthesis model is optimized to be higher. The feature values ​​of the initial speech feature vector are updated iteratively to obtain the target speech feature vector of the target subject; the text corresponding to the real speech is the same as the text corresponding to the speech; The target speech feature vector and the text to be synthesized are input into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

2. The speech synthesis method according to claim 1, wherein: The step of iteratively updating the feature values ​​of the initial speech feature vector to obtain the target speech feature vector of the target object includes: Acquire a first voice sample of the target object and a text sample corresponding to the first voice sample; For each update iteration of the feature value, the text sample and the latest initial speech feature vector are input into the speech synthesis model to obtain synthesized speech; Calculating a loss value of the speech synthesis model based on the first speech sample and the synthesized speech; the smaller the loss value, the higher the similarity between the first speech sample and the synthesized speech; The parameters of the speech synthesis model are fixed, and the loss value of the speech synthesis model is minimized as an optimization goal. The feature value of the latest initial speech feature vector is updated according to the loss value.

3. The speech synthesis method according to claim 1, wherein: The initial speech feature vector is obtained by the following steps: Performing feature fusion on preset speech feature vectors of preset speech samples of at least one preset object to obtain a fused speech feature vector; the preset object includes the target object; the preset speech feature vector and the fused speech feature vector include first feature values ​​of each preset speech feature dimension; Filtering at least one target speech feature dimension from all preset speech features according to the weights of the preset speech feature dimensions; The weight is used to measure the degree to which the preset speech feature dimension can reflect the unique speech feature of the target object; The first feature value corresponding to the target speech feature dimension is extracted from the fused speech feature vector to form an initial speech feature vector.

4. The speech synthesis method according to claim 3, wherein: The weight of each preset speech feature dimension to the target object is determined by the following steps: Extracting a sample speech feature vector corresponding to a second speech sample of the target object; the sample speech feature vector includes second feature values ​​of each preset speech feature dimension; Performing speech feature decomposition on each sample speech feature vector to obtain a second feature value of each sample speech feature vector in each preset speech feature dimension; For each preset speech feature dimension, a weight of the preset speech feature dimension to the target object is calculated according to a second feature value of the preset speech feature dimension in each sample speech feature vector.

5. The speech synthesis method according to claim 4, characterized in that The calculating, based on the second feature value of the preset speech feature dimension in each sample speech feature vector, a weight of the preset speech feature dimension for the target object includes: The variance of all second feature values ​​of the preset speech feature dimension is calculated to obtain the weight of the preset speech feature dimension to the target object.

6. The speech synthesis method according to claim 3, characterized in that: The step of selecting a target speech feature dimension from all preset speech features according to the weight of each preset speech feature dimension on the target object includes: Sort all preset speech feature dimensions by weighting the target object; A preset number of preset speech feature dimensions with the largest weights are determined as target speech feature dimensions.

7. The speech synthesis method according to claim 3, wherein: Before performing feature fusion on the preset speech feature vectors of the preset speech sample of at least one preset object to obtain the fused speech feature vector, the method further includes: Preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference are removed to obtain a final preset speech feature vector.

8. The speech synthesis method according to claim 7, characterized in that: The removing of the preset speech feature vectors whose difference from other preset speech feature vectors is greater than a preset difference comprises: Determining whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than a preset difference; If so, for each preset speech feature vector, count the number of other preset speech feature vectors whose difference from the preset speech feature vector is greater than the preset difference, to obtain the difference degree corresponding to the preset speech feature vector; After removing the preset speech feature vector with the largest difference, continue to determine whether there is a preset speech feature vector whose difference from other preset speech feature vectors is greater than the preset difference, until the differences between the preset speech feature vectors are all less than the preset difference.

9. The speech synthesis method according to claim 3, wherein: The step of inputting the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized includes: Inputting the text to be synthesized into the text encoder in the speech synthesis model to obtain a text vector to be synthesized; Inputting the target speech feature vector into the attention model in the speech synthesis model, so that the attention model generates an auxiliary speech feature vector corresponding to the auxiliary speech feature dimension based on the text vector to be synthesized, and the auxiliary speech feature dimension and the target speech feature dimension constitute the entire speech feature dimension; the attention model aligns the text vector to be synthesized with the target speech feature vector and the auxiliary speech feature vector; The correspondence between the aligned text vector to be synthesized, the target speech feature vector and the auxiliary speech feature vector is input into the decoder in the speech synthesis model to obtain the target speech of the text to be synthesized.

10. A speech synthesis device, characterized in that: The device includes: An acquisition module is used to obtain the text to be synthesized corresponding to the target object; An updating module, configured to fix parameters of the speech synthesis model, iteratively update the feature values ​​of the initial speech feature vector with the optimization goal of achieving a higher similarity between the real speech of the target subject and the speech of the target subject synthesized by the speech synthesis model, and obtain a target speech feature vector of the target subject; the text corresponding to the real speech is the same as the text corresponding to the speech; The synthesis module is used to input the target speech feature vector and the text to be synthesized into the speech synthesis model to synthesize the target speech corresponding to the text to be synthesized.

Images