Virtual sound synthesis method, apparatus, and related device
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- IFLYTEK CO LTD
- Filing Date
- 2022-11-08
- Publication Date
- 2026-07-24
Smart Images
Figure CN116469369B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of speech synthesis technology, and more specifically, to a virtual voice synthesis method, apparatus and related equipment. Background Technology
[0002] Speech synthesis technology is an intelligent voice technology that converts text into speech, and it is one of the core technologies for realizing human-computer interaction. With the continuous development and improvement of speech synthesis technology, it has now been widely applied to all aspects of social life, including public services (information broadcasting, intelligent customer service, etc.), smart hardware (smart speakers, smart robots, etc.), smart transportation (voice navigation, intelligent in-vehicle devices, etc.), education (smart classrooms, foreign language learning, etc.), and entertainment (audio reading, film dubbing, virtual IP, etc.), creating extensive economic and social value.
[0003] With the increasing application of speech synthesis, the demand for customized speech synthesis voice libraries, as well as the need for diversity and personalization of these libraries, is constantly rising. Existing virtual synthesized voice solutions generally involve: first, training a multi-person speech synthesis model, representing each speaker using a speaker encoding vector. This speaker encoding vector can be a learnable vector or a vector extracted from a pre-trained voiceprint recognition model. After the model training is complete, new virtual synthesized voices are constructed through interpolation and combination of the speaker encoding vectors.
[0004] Because speaker coding vectors are not necessarily continuous, the virtual synthesized voice obtained by interpolation in existing technologies is of poor quality. Summary of the Invention
[0005] In view of the above problems, this application is made to provide a virtual voice synthesis method, apparatus, and related equipment to solve the problem that existing methods for obtaining virtual synthesized voice through speaker encoding vectors often result in poor sound quality. The specific solution is as follows:
[0006] Firstly, a virtual voice synthesis method is provided, including:
[0007] Obtain a pre-constructed timbre feature space, which is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using a target generation model and mapping it through inverse transformation.
[0008] Samples are taken from the timbre feature space to obtain a sampled virtual timbre feature vector;
[0009] Virtual sound is synthesized based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound.
[0010] Secondly, a virtual voice synthesis device is provided, comprising:
[0011] The timbre feature space acquisition unit is used to acquire a pre-constructed timbre feature space, which is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using a target generation model and mapping it through inverse transformation.
[0012] A virtual timbre feature vector sampling unit is used to sample from the timbre feature space to obtain a sampled virtual timbre feature vector;
[0013] The virtual sound synthesis unit is used to synthesize virtual sound based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound.
[0014] Thirdly, an electronic device is provided, comprising: a memory and a processor;
[0015] The memory is used to store programs;
[0016] The processor is used to execute the program to implement the various steps of the virtual voice synthesis method described above.
[0017] Fourthly, a storage medium is provided on which a computer program is stored, which, when executed by a processor, implements the various steps of the virtual sound synthesis method described above.
[0018] By employing the above technical solution, this application, in order to generate a massive amount of virtual synthesized voices with different timbres, pre-processes a target generation model to perform distribution modeling on the original timbre feature vectors extracted from training speech of multiple speakers and then performs an inverse transformation mapping to obtain a smooth and continuous hidden layer space as the timbre feature space. The target generation model of this application is an invertible generative probabilistic model, which can map any complex probability distribution model to a smooth and continuous distribution space through an inverse transformation. Given this characteristic of the target generation model, the final timbre feature space is a smooth and continuous space. The virtual timbre feature vectors sampled in this space are continuous. Then, based on the text feature vector of the text to be synthesized and the virtual timbre feature vectors, virtual voice synthesis is performed to obtain the synthesized virtual voice. Compared to existing discontinuous speaker encoding vectors, this application, when performing virtual voice synthesis based on virtual timbre feature vectors sampled from a smooth and continuous timbre feature space, produces synthesized speech of better quality and higher robustness.
[0019] Meanwhile, the sampling target generation model of this application is based on the timbre feature space constructed from the training speech of multiple speakers, which contains a richer variety and number of timbre types, meeting the application requirements of virtual voice synthesis tasks for a rich synthetic sound library. Attached Figure Description
[0020] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings:
[0021] Figure 1 This is a flowchart illustrating the virtual voice synthesis method provided in an embodiment of this application;
[0022] Figure 2 This is a schematic diagram illustrating the timbre feature space construction process provided in the embodiments of this application;
[0023] Figure 3 An example of an optional structure diagram for a multi-speaker speech model is provided;
[0024] Figures 4a-4b Two other optional architecture diagrams for multi-speaker speech models are illustrated.
[0025] Figure 5 This is a schematic diagram of a virtual voice synthesis device provided in an embodiment of this application;
[0026] Figure 6 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation
[0027] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0028] This application provides a virtual voice synthesis method. First, it models the timbre feature space of speech based on a target generation model. Then, it synthesizes virtual voice based on the modeled timbre feature space. Using this method, sampling can be performed in a constructed smooth and continuous timbre feature space, and virtual voice synthesis can be performed based on the sampled virtual timbre feature vectors. This enables the instantaneous generation of a massive amount of synthesized speech with different timbres.
[0029] The proposed solution can be implemented based on a terminal with data processing capabilities, such as a mobile phone, computer, server, or cloud platform.
[0030] Next, combined Figure 1 The virtual voice synthesis method of this application may include the following steps:
[0031] Step S100: Obtain the pre-constructed timbre feature space. This space is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using the target generation model and mapping it through inverse transformation.
[0032] Specifically, this application can pre-collect training voice recordings from multiple speakers, with each speaker collecting at least several hundred sentences, each sentence lasting approximately ten seconds. The age distribution of the collected speaker data should be as balanced as possible. Age data can cover children's voices, teenagers' voices, young adults' voices, middle-aged voices, and elderly voices. Furthermore, for each age group, the number of male and female speakers should be balanced.
[0033] The training voice recordings of the speakers collected can be obtained through methods such as speaker recording and audio data collection.
[0034] For the collected training speech, the corresponding original timbre feature vector is obtained through timbre encoding.
[0035] Furthermore, a target generation model is used to model the distribution of each original timbre feature vector and then undergoes inverse transformation mapping to obtain a smooth and continuous hidden layer space, which is used as the timbre feature space.
[0036] The target generation model can employ a flow model, such as the Glow flow model or the NICE flow model. A flow model is a generative probabilistic model and is also an invertible model, capable of projecting any complex probability distribution model onto a smooth, continuous distribution space through the inverse transformation of the flow model. Taking the Glow flow model as an example, it can project any complex probability distribution model onto a standard normal distribution space through the inverse transformation of the flow model.
[0037] Step S110: Sample from the timbre feature space to obtain the sampled virtual timbre feature vector.
[0038] Specifically, during timbre feature space sampling, random sampling can be used to obtain virtual timbre feature vectors. Since the timbre feature space is smoothly and continuously distributed, a massive number of virtual timbre feature vectors can be sampled instantly, which facilitates the subsequent generation of a large number of synthesized virtual sounds.
[0039] Step S120: Perform virtual sound synthesis based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound.
[0040] Specifically, this application can feed the text feature vector of the text to be synthesized and the virtual timbre feature vector into a trained multi-speaker speech synthesis model to obtain the virtual voice synthesized by the model.
[0041] Optionally, during speech synthesis, in addition to inputting the text to be synthesized and the virtual timbre feature vector, the user can further select a target speaker representation vector. This target speaker representation vector, along with the text to be synthesized and the virtual timbre feature vector, is then input into the multi-speaker speech synthesis model. The target speaker representation vector can be a vector representing the prosodic style of different speakers. Therefore, the user can select target speaker representation vectors with similar prosodic styles from the training set based on the prosodic style of the virtual voice to be synthesized.
[0042] The virtual voice synthesis method provided in this application, in order to generate a massive number of virtual synthesized voices with different timbres, first uses a target generation model to model the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers and then performs an inverse transformation mapping to obtain a smooth and continuous hidden layer space as the timbre feature space. The target generation model of this application is an invertible generative probabilistic model, which can map any complex probability distribution model to a smooth and continuous distribution space through an inverse transformation. Given this characteristic of the target generation model, the final timbre feature space is a smooth and continuous space. The virtual timbre feature vectors sampled in this space are continuous. Then, based on the text feature vector of the text to be synthesized and the virtual timbre feature vectors, virtual voice synthesis is performed to obtain the synthesized virtual voice. Compared with the existing discontinuous speaker encoding vectors, when virtual voice synthesis is performed based on the virtual timbre feature vectors sampled from the smooth and continuous timbre feature space, the synthesized speech obtained in this application has better quality and higher robustness.
[0043] Meanwhile, the sampling target generation model of this application is based on a timbre feature space constructed from the training speech of multiple speakers, which contains a richer variety and number of timbre types, meeting the application requirements of virtual voice synthesis tasks for a rich synthetic voice library. Using the virtual voice synthesis method of this application, a completely new virtual voice can be synthesized, or an existing voice can be replicated.
[0044] In some embodiments of this application, the construction process of the timbre feature space described in the above embodiments is introduced, such as... Figure 2 As shown, the process may include:
[0045] Step S200: Obtain a training set consisting of training speech, training text, and speaker representation vectors from multiple speakers.
[0046] Specifically, in order to model the speech timbre feature space, this case collected a massive amount of speaker training data for modeling. The collected training data included the speaker's training speech, the corresponding training text, and the speaker's representation vector.
[0047] The speaker's training speech can be clean speech data, and the training text is the pronunciation text information corresponding to the clean speech data.
[0048] Speaker representation vectors are used to model and differentiate the prosody of different speakers. Speaker representation vectors can take various forms; for example, one-hot encoding can be used to encode each speaker's representation vector. Alternatively, a trained speaker recognition model can extract the voiceprint representation of each speaker's training speech as their representation vector, and so on.
[0049] Furthermore, to ensure the model's effectiveness, the training data collected in this step can include data from at least a thousand speakers. Each speaker can have at least a hundred sentences collected, with each sentence lasting approximately ten seconds. The age distribution of the collected speakers should be as balanced as possible. Age ranges can be covered, including children's voices, teenagers' voices, young adults' voices, middle-aged voices, and elderly voices. Additionally, within each age group, the number of male and female speakers should be balanced.
[0050] The training voice recordings of the speakers collected can be obtained through methods such as speaker recording and audio data collection.
[0051] Step S210: Train a multi-speaker speech model using the training set. The multi-speaker speech model includes a target generation model that can model the speaker timbre feature space.
[0052] Specifically, the multi-speaker speech model designed in this application can be used for:
[0053] Based on the speaker's training text and the speaker's representation vector, a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre is extracted. The speaker's training speech is timbre-encoded to obtain an original timbre feature vector. This original timbre feature vector is then inversely transformed and mapped to a smooth, continuous hidden layer space using a target generation model, resulting in a mapped timbre hidden layer feature vector. Decoding is then performed based on this timbre hidden layer feature vector and the target acoustic representation vector to obtain the decoded acoustic information.
[0054] As shown above, the multi-speaker speech model designed in this application separates and models the speaker's prosodic style and timbre during the modeling process. Specifically, it extracts target acoustic representation vectors that are related to the speaker's prosodic duration but independent of timbre, and uses a target generation model to inversely transform the original timbre feature vector of the training speech to a smooth and continuous hidden layer space, obtaining the timbre hidden layer feature vector. Finally, based on the timbre hidden layer feature vector and the target acoustic representation vector, decoding is performed to obtain the acoustic information.
[0055] In this embodiment, the target generation model in the multi-speaker speech model can employ a streaming model, such as the Glow streaming model or the NICE streaming model. A streaming model is a generative probabilistic model and is also an invertible model, capable of projecting any complex probability distribution model onto a smooth, continuous distribution space through the inverse transformation of the streaming model. Taking the Glow streaming model as an example, it can project any complex probability distribution model onto a standard normal distribution space through the inverse transformation of the streaming model.
[0056] Among them, acoustic information can be various types of acoustic information such as original waveform, linear spectrum, Mel spectrum, etc.
[0057] Step S220: Calculate the acoustic loss and the loss of the target generation model, and train the network parameters of the multi-speaker speech model.
[0058] Specifically, the acoustic loss can be calculated based on the decoded acoustic information and the speaker's training speech.
[0059] Taking acoustic information as the spectral feature y as an example, the acoustic loss can be expressed as L mel :
[0060]
[0061] Where T represents the number of frames contained in the current training audio, and y i and These are the actual spectral features and the model-predicted spectral features of the i-th frame of the current training speech, respectively.
[0062] Further, the loss of the target generation model is calculated.
[0063] Taking the Glow flow model as an example, the log-likelihood loss function L of the Glow flow model can be calculated. Glow :
[0064] L Glow =log p(z;N(0,1))+log(|det(f G )| -1 )
[0065] Where z is the timbre hidden layer feature vector obtained after the inverse transformation of the current training speech using the Glow stream model, N(0,1) represents the standard normal distribution, and det(f G ) is the Jacobian determinant of the Glow flow model.
[0066] After calculating the acoustic loss and the target generation model loss as described above, the total loss L is obtained by combining the acoustic loss and the target generation model loss.
[0067] L = L mel +λ Glow ·L Glow
[0068] Where, λ Glow The weights of the loss function in the Glow flow model are negative.
[0069] The network parameters of the multi-speaker speech model are trained with the goal of minimizing the total loss L, until the set training termination condition is met. At this point, the standard normal distribution followed by the hidden layer space of the Glow stream model is the obtained timbre feature space.
[0070] Step S230: Use the hidden space of the target generation model in the trained multi-speaker speech model as the timbre feature space.
[0071] Specifically, after training the multi-speaker speech model as described above, the hidden space of the target generation model can be used as the final timbre feature space.
[0072] The timbre feature space construction method provided in this application directly models the timbre feature space. Traditional speech synthesis models typically use speaker representation vectors to distinguish different speaker data. However, since different speakers have different prosodic styles and timbres, the speaker representation vector contains comprehensive information about both prosodic style and timbre. To learn the timbre feature space distribution of speech, unlike traditional speech synthesis models, this application's multi-speaker speech model can extract a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre, based on the speaker's training text and representation vector. It further obtains the original timbre feature vector by timbre encoding the speaker's training speech, and uses a target generation model to inversely transform and map the original timbre feature vector to a smooth and continuous hidden layer space, obtaining the timbre hidden layer feature vector. Decoding is then performed based on the timbre hidden layer feature vector and the target acoustic representation vector to obtain the decoded acoustic information. In other words, this application's multi-speaker speech model decouples and separates the speaker's prosodic style and timbre during the decoding process. Furthermore, by employing a target generation model, this application can inversely transform and map the original timbre feature vector to a smooth and continuous hidden layer space. By using the hidden layer space of the target generation model in the trained multi-speaker speech model as the timbre feature space, it can significantly improve the efficiency of speech synthesis library construction, enrich the timbre types and number of speech synthesis library, and meet the application needs of virtual humans, smart hardware and other businesses for rich synthesized sound libraries.
[0073] Meanwhile, the target generation model of this application is a generative probability model, which can map any complex probability distribution model to a smooth and continuous distribution space through inverse transformation. Given this characteristic of the target generation model, the final timbre feature space is a smooth and continuous space. The timbre features sampled in this space are continuous. Compared with the existing discontinuous speaker coding vectors, when the timbre features sampled from the smooth and continuous timbre feature space of this application are used for virtual voice synthesis, the synthesized speech has better quality and higher robustness.
[0074] Combination Figure 3 As shown, this embodiment provides an optional component structure for a multi-speaker speech model.
[0075] Multi-speaker speech models may include:
[0076] The system includes an acoustic representation vector extraction module, a timbre space learning module, and a decoding module, wherein the timbre space learning module includes a timbre encoding module and a target generation model.
[0077] The acoustic representation vector extraction module is used to extract target acoustic representation vectors that are related to the speaker's prosodic duration and independent of the speaker's timbre, based on the speaker's training text and the speaker's representation vectors.
[0078] Specifically, the inputs to the acoustic representation vector extraction module may include the text features x of the speaker's training text and the speaker's representation vector s. The text features x may include information such as phonemes, tones, and prosodic levels.
[0079] The acoustic representation vector extraction module can extract a target acoustic representation vector h1 that is related to the speaker's prosodic duration and independent of timbre based on the input features.
[0080] The input speaker representation vector s is used to control the extracted target acoustic representation vector h1 to be related to the speaker's prosodic duration. Furthermore, by designing the structure of the acoustic representation vector extraction module, it can be ensured that the extracted target acoustic representation vector h1 is independent of the speaker's timbre, thereby ensuring that the timbre space learning module can learn the decoupled timbre features.
[0081] The target acoustic representation vector h1 is fed into the decoding module to achieve prosodic control decoding.
[0082] The structure of the acoustic representation vector extraction module will be described in detail in subsequent embodiments of this application.
[0083] The timbre encoding module is used to encode the timbre of the speaker's training speech to obtain the original timbre feature vector.
[0084] Specifically, the input to the timbre encoding module can be the acoustic features y of the speaker's training speech, which can be waveform features, spectral features, etc.
[0085] The timbre encoding module can be initialized randomly or a pre-trained speaker classification model, etc. No special constraints are imposed in this embodiment.
[0086] It should be noted that during a training process, the training data input to the acoustic representation vector extraction module and the data input to the timbre encoding module only need to be training data from the same speaker; it is not required that the input training speech and training text be paired.
[0087] The timbre encoding module extracts the original timbre feature vector h2 by performing operations such as convolution and pooling on the acoustic features y of the input training speech.
[0088] The target generation model is used to inversely transform the original timbre feature vector and map it to a smooth and continuous hidden layer space to obtain the mapped timbre hidden layer feature vector.
[0089] Specifically, the input to the target generation model is the original timbre feature vector h2 output by the timbre encoding module. After inverse transformation by the target generation model, it is mapped to the hidden layer space to obtain the inversely transformed timbre hidden layer feature vector z. The timbre hidden layer feature vector z is then fed into the decoding module to achieve timbre-controlled decoding.
[0090] The decoding module is used to decode based on the timbre hidden layer feature vector and the target acoustic representation vector to obtain the decoded acoustic information.
[0091] The multi-speaker speech model provided in this embodiment can achieve separate modeling of prosody and timbre. At the same time, the hidden space obtained by modeling the target generation model in the trained multi-speaker speech model serves as the timbre feature space, ensuring the smoothness and continuity of the timbre feature space.
[0092] Based on the structure of the multi-speaker speech model described above, this application embodiment introduces the aforementioned step S120, which is an optional implementation method for synthesizing virtual sound based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound.
[0093] Specifically, the aforementioned multi-speaker speech model can be used as the speech synthesis model for virtual voice synthesis. That is, the text feature vector of the text to be synthesized, the selected target speaker representation vector, and the virtual timbre feature vector can be input into the trained multi-speaker speech model to obtain the target acoustic information decoded by the multi-speaker speech model. Further, based on the target acoustic information, the synthesized virtual voice is obtained.
[0094] When using a trained multi-speaker speech model for virtual voice synthesis, there is no need to input speaker training speech as in the training phase. The input virtual timbre feature vector can be directly sent to the decoding module. At the same time, the acoustic representation vector extraction module encodes the text feature vector of the input text to be synthesized and the selected target speaker representation vector to obtain the target acoustic representation vector that is independent of the speaker's timbre. Finally, the decoding module decodes the target acoustic information based on the target speaker representation vector and the input virtual timbre feature vector.
[0095] Target acoustic information can take many forms, such as waveform features, spectral features, or directly synthesized speech data. Based on this target acoustic information, synthesized speech can be obtained.
[0096] Taking the target acoustic information as a spectral feature as an example, the spectral feature can be input into a vocoder to obtain synthesized speech.
[0097] This embodiment utilizes a trained multi-speaker speech model to synthesize a massive amount of virtual voices. Since the timbre feature space is a smooth and continuous space, the timbre features sampled within this space are continuous. Compared to existing discontinuous speaker coding vectors, this application achieves better quality and higher robustness in synthesizing virtual voices based on timbre features sampled from a smooth and continuous timbre feature space. Furthermore, by sampling within a continuous timbre feature space, it is possible to synthesize a completely new virtual voice or a replica of an existing voice.
[0098] In some embodiments of this application, several optional implementation methods of the acoustic representation vector extraction module described above are introduced.
[0099] The first type
[0100] Combination Figure 4a As shown, the acoustic representation vector extraction module may include: a text encoding module, a duration prediction module, and a duration adjustment module.
[0101] The text encoding module is used to encode the text features x of the input speaker's training text to obtain the text encoding vector h0.
[0102] In this context, the text features x of the training text can be phoneme-level text features, and the text encoding vector obtained after encoding also corresponds to the phoneme-level text encoding vector h0.
[0103] The duration prediction module is used to predict the duration information d of the training text based on the text encoding vector h0 and the input speaker representation vector s.
[0104] Specifically, the duration information of the training text can be the duration d of each phoneme in the training text. The duration prediction module is used to learn the duration d of each phoneme in the input text. The speaker representation vector s input to the duration prediction module is used to control the modeling of the duration of different speakers.
[0105] The duration adjustment module is used to expand the text encoding vector according to the duration information of the training text to obtain a target acoustic representation vector that is related to the speaker's prosodic duration and is independent of the speaker's timbre.
[0106] Specifically, the text encoding vector h0 obtained by the text encoding module does not contain the speaker's prosodic information. In order to obtain the target acoustic representation vector related to the speaker's prosodic duration, the duration adjustment module expands the text encoding vector h0 according to the duration information d of the training text output by the duration prediction module, and then obtains the target acoustic representation vector h1 that is related to the speaker's prosodic duration and is unrelated to the speaker's timbre.
[0107] Taking the duration information of the training text, including the duration of each phoneme in the training text, as an example, the process by which the duration adjustment module expands the text encoding vector according to the duration information of the training text to obtain the target acoustic representation vector can include:
[0108] The duration adjustment module expands the phoneme-level text encoding vector h0 according to the duration d of each phoneme to obtain the target acoustic representation vector h1.
[0109] There are several ways to expand the text encoding vector according to the phoneme duration. For example, the vector h corresponding to each phoneme in the text encoding vector h0 can be expanded in several ways. 0j If the duration of the j-th phoneme is n frames, then the vector h corresponding to that phoneme is... 0j The target acoustic representation vector h1 is copied n times and expanded. The length of the expanded vector h1 is the same as the sum of the durations of the phonemes in the training text.
[0110] Of course, the method of expanding the text encoding vector according to the phoneme duration is not limited to the above copying form. Other expansion forms can also be used, which will not be elaborated here.
[0111] by Figure 4a Taking the multi-speaker speech model shown as an example, since the acoustic representation vector extraction module includes a duration prediction module, this duration prediction module can be trained independently or jointly with the multi-speaker speech model. When the duration prediction module is jointly trained with the multi-speaker speech model, the total loss function during the training of the multi-speaker speech model can further include the duration prediction loss L. dur :
[0112] Duration prediction loss L dur It can be calculated based on the predicted duration information of the training text and the actual duration information of the labeled training text.
[0113]
[0114] Where P is the number of phonemes contained in the current training text. j and These are the actual and predicted duration frames for the j-th phoneme in the current training text, respectively.
[0115] The final total loss L is:
[0116] L = L mel +L dur +λ Glow ·L Glow
[0117] The second type
[0118] Combination Figure 4b As shown, the acoustic representation vector extraction module may include: a feature extraction module that implicitly models duration based on an attention mechanism.
[0119] This feature extraction module uses an attention mechanism to predict the target acoustic representation vector h1, which is related to the speaker's prosodic duration but independent of the speaker's timbre, based on the text features x of the speaker's training text and the speaker's representation vector s.
[0120] and Figure 4a The structure of the example acoustic representation vector extraction module differs from that of the example module. Figure 4b The feature extraction module of the example uses an attention mechanism to predict the target acoustic representation vector and stop symbols that are related to the speaker's prosodic duration but not to the speaker's timbre. By predicting the stop symbols, the length of the target acoustic representation vector is controlled, thus implicitly modeling the duration, and finally obtaining the target acoustic representation vector that is related to the speaker's prosodic duration but not to the speaker's timbre.
[0121] by Figure 4b Taking the multi-speaker speech model shown as an example, since the acoustic representation vector extraction module includes a feature extraction module that implicitly models duration based on an attention mechanism, and this feature extraction module can be jointly trained with the multi-speaker speech model, the total loss function during the training of the multi-speaker speech model can further include the training loss L of the feature extraction module. att :
[0122] L att It can be calculated based on the predicted target acoustic representation vector and the real acoustic representation vector independent of speaker timbre labeled in the training text, as well as based on the predicted stop symbols and the real stop symbols labeled in the training text.
[0123]
[0124] Where T is the number of frames contained in the training speech corresponding to the current training text. i and Let be the real and predicted target acoustic representation vectors, independent of speaker timbre, of the i-th frame in the training speech, respectively. and I i denoted as the posterior probability of the predicted stop symbol and the actual stop symbol in the i-th frame of the training speech, respectively, and CE(·) is the cross-entropy.
[0125] The final total loss L is:
[0126] L = L mel +L att +λ Glow ·L Glow
[0127] The virtual voice synthesis apparatus provided in the embodiments of this application is described below. The virtual voice synthesis apparatus described below can be referred to in correspondence with the virtual voice synthesis method described above.
[0128] See Figure 5 , Figure 5 This is a schematic diagram of the structure of a virtual voice synthesis device disclosed in an embodiment of this application.
[0129] like Figure 5 As shown, the device may include:
[0130] The timbre feature space acquisition unit 11 is used to acquire a pre-constructed timbre feature space, which is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using a target generation model and mapping it through inverse transformation.
[0131] The virtual timbre feature vector sampling unit 12 is used to sample from the timbre feature space to obtain the sampled virtual timbre feature vector;
[0132] The virtual sound synthesis unit 13 is used to synthesize virtual sound based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound.
[0133] Optionally, the apparatus of this application may further include a timbre feature space construction unit for constructing a timbre feature space, the timbre feature space construction unit may include:
[0134] The training set acquisition unit is used to acquire a training set consisting of training speech, training text, and speaker representation vectors from multiple speakers.
[0135] The model training unit is used to train a multi-speaker speech model using the training set. The model is configured to: extract a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre, based on the speaker's training text and the speaker's representation vector; encode the speaker's training speech for timbre to obtain an original timbre feature vector; inversely transform the original timbre feature vector to a smooth, continuous hidden layer space using a target generation model to obtain a mapped timbre hidden layer feature vector; decode the timbre hidden layer feature vector and the target acoustic representation vector to obtain decoded acoustic information; calculate the acoustic loss based on the decoded acoustic information and the speaker's training speech; calculate the loss of the target generation model; combine the acoustic loss and the loss of the target generation model to obtain a total loss; and train the network parameters of the multi-speaker speech model with minimizing the total loss as the training objective.
[0136] The timbre feature space creation unit is used to take the hidden layer space of the target generation model in the trained multi-speaker speech model as the timbre feature space.
[0137] The multi-speaker speech model may include: an acoustic representation vector extraction module, a timbre space learning module, and a decoding module. The timbre space learning module includes a timbre encoding module and a target generation model.
[0138] The acoustic representation vector extraction module is used to extract target acoustic representation vectors that are related to the speaker's prosodic duration and unrelated to the speaker's timbre, based on the speaker's training text and the speaker's representation vectors.
[0139] The timbre encoding module is used to encode the speaker's training speech to obtain the original timbre feature vector;
[0140] The target generation model is used to inversely transform and map the original timbre feature vector to a smooth and continuous hidden layer space to obtain the mapped timbre hidden layer feature vector.
[0141] The decoding module is used to decode based on the timbre hidden layer feature vector and the target acoustic representation vector to obtain the decoded acoustic information.
[0142] This application provides two optional structures for the acoustic representation vector extraction module:
[0143] The first type of acoustic representation vector extraction module may include: a text encoding module, a duration prediction module, and a duration adjustment module;
[0144] The text encoding module is used to encode the text features of the input speaker's training text to obtain a text encoding vector;
[0145] The duration prediction module is used to predict the duration information of the training text based on the text encoding vector and the input speaker representation vector.
[0146] The duration adjustment module is used to expand the text encoding vector according to the duration information of the training text to obtain a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre. Based on this, the total loss during multi-speaker speech model training may further include:
[0147] The duration prediction loss is calculated based on the predicted duration information of the training text and the actual duration information of the labeled training text.
[0148] Optionally, the duration information of the training text may include the duration of each phoneme in the training text. Based on this, the process by which the duration adjustment module expands the text encoding vector according to the duration information of the training text to obtain the target acoustic representation vector may include:
[0149] The duration adjustment module expands the phoneme-level text encoding vector according to the duration of each phoneme to obtain the target acoustic representation vector.
[0150] The second type, the acoustic representation vector extraction module, may include: a feature extraction module based on attention mechanism for implicit duration modeling. This feature extraction module, based on the text features of the input speaker's training text and the speaker's representation vector, uses an attention mechanism to predict the target acoustic representation vector and stop symbols that are related to the speaker's prosodic duration but independent of the speaker's timbre. Furthermore, the total loss during multi-speaker speech model training may also include:
[0151] The training loss of the feature extraction module is calculated based on the predicted target acoustic representation vector and the real acoustic representation vector independent of speaker timbre of the training text annotation, as well as the predicted stop symbols and the real stop symbols of the training text annotation.
[0152] Optionally, the speaker representation vector acquired by the training set acquisition unit mentioned above may include:
[0153] One-hot encoding is used to encode each speaker to obtain the speaker's representation vector;
[0154] or,
[0155] The trained speaker recognition model is used to extract the voiceprint representation of each speaker's training speech, which is then used as the representation vector for each speaker.
[0156] Optionally, the target generation model included in the above multi-speaker speech model can be a stream model, and the hidden space of the corresponding target generation model is a standard normal distribution space.
[0157] The virtual voice synthesis device provided in this application embodiment can be applied to electronic devices, such as terminals: mobile phones, computers, etc. Optionally, Figure 6 A hardware block diagram of the electronic device is shown, with reference to... Figure 6 The hardware structure of an electronic device may include: at least one processor 1, at least one communication interface 2, at least one memory 3, and at least one communication bus 4;
[0158] In this embodiment of the application, the number of processor 1, communication interface 2, memory 3, and communication bus 4 is at least one, and processor 1, communication interface 2, and memory 3 communicate with each other through communication bus 4;
[0159] Processor 1 may be a central processing unit (CPU), an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement embodiments of the present invention.
[0160] Memory 3 may include high-speed RAM, and may also include non-volatile memory, such as at least one disk storage device;
[0161] The memory stores a program, which the processor can call to implement the various steps of the aforementioned virtual sound synthesis method.
[0162] Optionally, the refined and extended functions of the program can be found in the description above.
[0163] This application embodiment also provides a storage medium that can store a program suitable for execution by a processor, the program being used to implement the various steps of the aforementioned virtual sound synthesis method.
[0164] Optionally, the refined and extended functions of the program can be found in the description above.
[0165] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0166] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.
[0167] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A method for synthesizing virtual voice, characterized in that, include: Obtain a pre-constructed timbre feature space, which is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using a target generation model and mapping it through inverse transformation. Random sampling is performed from the timbre feature space to obtain the sampled virtual timbre feature vector; Virtual sound is synthesized based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound; The process of constructing the timbre feature space includes: Obtain a training set consisting of training speech, training text, and speaker representation vectors from multiple speakers; A multi-speaker speech model is trained using the training set, wherein the model is used to: extract a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre based on the speaker's training text and the speaker's representation vector; encode the speaker's training speech for timbre to obtain an original timbre feature vector; use a target generation model to inversely transform and map the original timbre feature vector to a smooth and continuous hidden layer space to obtain a mapped timbre hidden layer feature vector; and decode based on the timbre hidden layer feature vector and the target acoustic representation vector to obtain decoded acoustic information. Based on the decoded acoustic information and the speaker's training speech, the acoustic loss is calculated, the loss of the target generation model is calculated, and the total loss is obtained by combining the acoustic loss and the loss of the target generation model. The network parameters of the multi-speaker speech model are trained with the goal of minimizing the total loss. The hidden space of the target generation model in the trained multi-speaker speech model is used as the timbre feature space.
2. The method according to claim 1, characterized in that, The multi-speaker speech model includes: an acoustic representation vector extraction module, a timbre space learning module, and a decoding module. The timbre space learning module includes a timbre encoding module and a target generation model. The acoustic representation vector extraction module is used to extract target acoustic representation vectors that are related to the speaker's prosodic duration and unrelated to the speaker's timbre, based on the speaker's training text and the speaker's representation vectors. The timbre encoding module is used to encode the speaker's training speech to obtain the original timbre feature vector; The target generation model is used to inversely transform and map the original timbre feature vector to a smooth and continuous hidden layer space to obtain the mapped timbre hidden layer feature vector. The decoding module is used to decode based on the timbre hidden layer feature vector and the target acoustic representation vector to obtain the decoded acoustic information.
3. The method according to claim 2, characterized in that, The acoustic representation vector extraction module includes: a text encoding module, a duration prediction module, and a duration adjustment module; The text encoding module is used to encode the text features of the input speaker's training text to obtain a text encoding vector; The duration prediction module is used to predict the duration information of the training text based on the text encoding vector and the input speaker representation vector. The duration adjustment module is used to expand the text encoding vector according to the duration information of the training text to obtain a target acoustic representation vector that is related to the speaker's prosodic duration and is independent of the speaker's timbre. The total loss also includes: The duration prediction loss is calculated based on the predicted duration information of the training text and the actual duration information of the labeled training text.
4. The method according to claim 2, characterized in that, The acoustic representation vector extraction module includes: a feature extraction module for implicitly modeling duration based on an attention mechanism; The feature extraction module is used to predict the target acoustic representation vector and stop symbols that are related to the speaker's prosodic duration and independent of the speaker's timbre, based on the text features of the input speaker's training text and the speaker's representation vector, using an attention mechanism. The total loss also includes: The training loss of the feature extraction module is calculated based on the predicted target acoustic representation vector and the real acoustic representation vector independent of speaker timbre of the training text annotation, as well as the predicted stop symbols and the real stop symbols of the training text annotation.
5. The method according to claim 3, characterized in that, The duration information of the training text includes: the duration of each phoneme in the training text; The process by which the duration adjustment module expands the text encoding vector according to the duration information of the training text to obtain the target acoustic representation vector includes: The duration adjustment module expands the phoneme-level text encoding vector according to the duration of each phoneme to obtain the target acoustic representation vector.
6. The method according to any one of claims 1-5, characterized in that, The speaker's representation vector includes: One-hot encoding is used to encode each speaker to obtain the speaker's representation vector; or, The trained speaker recognition model is used to extract the voiceprint representation of each speaker's training speech, which is then used as the representation vector for each speaker.
7. The method according to any one of claims 1-5, characterized in that, The target generation model is a flow model, and the hidden space of the target generation model is a standard normal distribution space.
8. The method according to claim 1, characterized in that, The virtual sound synthesis based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound includes: The text feature vector of the text to be synthesized, the selected target speaker representation vector, and the virtual timbre feature vector are input into the trained multi-speaker speech model to obtain the target acoustic information after the multi-speaker speech model is decoded. Based on the target acoustic information, a synthesized virtual sound is obtained.
9. A virtual voice synthesis device, characterized in that, include: Timbre feature space construction unit, used to construct timbre feature space; The timbre feature space acquisition unit is used to acquire a pre-constructed timbre feature space, which is a smooth and continuous hidden layer space obtained by modeling the distribution of the original timbre feature vectors extracted from the training speech of multiple speakers using a target generation model and mapping it through inverse transformation. A virtual timbre feature vector sampling unit is used to randomly sample from the timbre feature space to obtain a sampled virtual timbre feature vector; A virtual sound synthesis unit is used to synthesize virtual sound based on the text feature vector of the text to be synthesized and the virtual timbre feature vector to obtain the synthesized virtual sound. The timbre feature space construction unit includes: The training set acquisition unit is used to acquire a training set consisting of training speech, training text, and speaker representation vectors from multiple speakers. The model training unit is used to train a multi-speaker speech model using the training set. The model is configured to: extract a target acoustic representation vector that is related to the speaker's prosodic duration and independent of the speaker's timbre, based on the speaker's training text and the speaker's representation vector; encode the speaker's training speech for timbre to obtain an original timbre feature vector; inversely transform the original timbre feature vector to a smooth, continuous hidden layer space using a target generation model to obtain a mapped timbre hidden layer feature vector; decode the timbre hidden layer feature vector and the target acoustic representation vector to obtain decoded acoustic information; calculate the acoustic loss based on the decoded acoustic information and the speaker's training speech; calculate the loss of the target generation model; combine the acoustic loss and the loss of the target generation model to obtain a total loss; and train the network parameters of the multi-speaker speech model with minimizing the total loss as the training objective. The timbre feature space creation unit is used to take the hidden layer space of the target generation model in the trained multi-speaker speech model as the timbre feature space.
10. An electronic device, characterized in that, include: Memory and processor; The memory is used to store programs; The processor is configured to execute the program to implement each step of the virtual sound synthesis method as described in any one of claims 1 to 8.
11. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements each step of the virtual sound synthesis method as described in any one of claims 1 to 8.