Speech synthesis method and device, equipment and storage medium
By adding a speech encoding module to the acoustic model of the speech synthesis system, using phoneme-level vectors of text and speech, the shortcomings of the existing speech synthesis system in terms of pronunciation effects are solved, and a more natural and smooth speech synthesis is achieved.
Patent Information
- Application Number
- CN202311393985.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-10-25
- Publication Date
- 2025-04-25
AI Technical Summary
The existing speech synthesis system has shortcomings in the pronunciation effect of synthesizing speech, mainly because the acoustic model only uses phoneme vectors of text and fails to make full use of phoneme vectors of speech.
By adding a speech encoding module to the traditional acoustic model, the acoustic model can utilize phoneme-level vectors of text and speech at the same time. The specific steps include obtaining the text phoneme sequence of the target text, using the trained acoustic model for feature conversion, generating text phoneme vectors and speech phoneme vectors, and synthesizing the speech corresponding to the target text.
By simultaneously utilizing phoneme-level vectors of text and speech, the rhythmic effect of synthetic speech is improved and the sound quality of the speech synthesis system is improved.
Smart Images
Figure CN119889284A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech synthesis technology, and in particular to a speech synthesis method, apparatus, device and storage medium. Background Art
[0002] With the continuous development of artificial intelligence technology, speech synthesis technology has been widely used. Speech synthesis is a technology that generates artificial speech through mechanical and electronic methods. It can convert text information generated by the computer itself or input externally into understandable and fluent speech. At present, traditional speech synthesis systems usually use acoustic models to extract the acoustic information of speech to form a spectrogram. However, the existing acoustic models only use the phoneme vectors of the text, and the rhythmic effect of the synthesized speech is low. Summary of the invention
[0003] In view of this, the embodiments of the present application provide a speech synthesis method, apparatus, device and storage medium, which can simultaneously utilize the phoneme-level vectors of text and speech to improve the rhythmic effect of synthesized speech.
[0004] The first aspect of an embodiment of the present application provides a speech synthesis method, comprising: obtaining a text phoneme sequence corresponding to a target text, wherein the target text is text to be converted into speech; using a trained acoustic model to perform feature conversion processing on the text phoneme sequence to obtain a text phoneme vector, and generating a corresponding speech phoneme vector based on the text phoneme vector, and generating an acoustic feature corresponding to the target text based on the text phoneme vector and the speech phoneme vector; and synthesizing the speech corresponding to the target text based on the acoustic feature.
[0005] In a possible implementation, before the step of using a trained acoustic model to perform feature conversion processing on the text phoneme sequence, it also includes: building an acoustic model in combination with a speech coding module, and jointly training the speech coding module when training the acoustic model using training sample data to obtain a trained acoustic model, and the trained acoustic model is used to generate acoustic features based on the text phoneme sequence.
[0006] In a possible implementation, the step of building an acoustic model in combination with a speech coding module and jointly training the speech coding module when training the acoustic model with training sample data to obtain a trained acoustic model includes: building an acoustic model, wherein the acoustic model includes a text coding module, a speech coding module, a variable-length prediction module, and a decoding module; constructing training sample data for training the acoustic model, wherein the training sample data includes a text phoneme sequence, a speech phoneme sequence, and standard acoustic features of the training text, wherein the speech phoneme sequence is used to obtain a standard speech phoneme vector of the training sample; and inputting the text phoneme sequence in the training sample into the The text encoding module is input to obtain a predicted text phoneme vector; the text phoneme vector is input to the speech encoding module to obtain a predicted speech phoneme vector; the predicted text phoneme vector and the predicted speech phoneme vector are combined and input to the variable-length prediction module to obtain a frame-level vector; the frame-level vector is input to the decoding module to obtain a predicted acoustic feature; the parameters of the speech encoding module are updated according to the predicted speech phoneme vector and the standard speech phoneme vector, and the parameters of the decoding module are updated according to the predicted acoustic feature and the standard acoustic feature, until the acoustic model is trained to a convergence state, so as to obtain a trained acoustic model.
[0007] In a possible implementation, the step of updating the parameters of the speech encoding module according to the predicted speech phoneme vector and the standard speech phoneme vector, and updating the parameters of the decoding module according to the predicted acoustic features and the standard acoustic features until the acoustic model is trained to a convergence state, includes: obtaining the loss function of the speech encoding module and the loss function of the acoustic model; using the loss function of the speech encoding module to calculate a first loss value between the predicted speech phoneme vector and the standard speech phoneme vector, and judging whether the first loss value meets a preset first threshold condition, and using the loss function of the acoustic model to calculate a second loss value between the predicted acoustic feature and the standard acoustic feature, and judging whether the second loss value meets a preset second threshold condition; if the first loss value meets the preset first threshold condition and the second loss value meets the preset second threshold condition, it is determined that the acoustic model is trained to a convergence state.
[0008] In one possible implementation, the step of obtaining the standard speech phoneme vector of the training sample includes: using an unsupervised speech recognition model to perform speech recognition on the speech phoneme sequence of the training sample to obtain multiple speech latent vectors corresponding to each phoneme in the speech phoneme sequence; for each phoneme, taking the mean of the multiple speech latent vectors as the vector representation of the phoneme; and concatenating the vector representations corresponding to all phonemes in the speech phoneme sequence to obtain the standard speech phoneme vector of the training sample.
[0009] In a possible implementation, the step of obtaining a text phoneme sequence corresponding to a target text includes: obtaining a target text to be converted into speech; performing word splitting processing on the target text to obtain a plurality of words; for each word, determining the language type of the word, and obtaining a phoneme table matching the word according to the language type; obtaining a text phoneme sequence corresponding to the word according to the phoneme table; and concatenating the text phoneme sequences corresponding to all the words in order of their positions in the target text to obtain a text phoneme sequence corresponding to the target text.
[0010] In a possible implementation, after the step of performing word splitting processing on the target text to obtain a plurality of words, the method further includes: performing data cleaning processing on the plurality of words.
[0011] The second aspect of an embodiment of the present application provides a speech synthesis device, including: an acquisition module, used to acquire a text phoneme sequence corresponding to a target text, wherein the target text is text to be converted into speech; a conversion module, used to perform feature conversion processing on the phoneme sequence using a trained acoustic model to obtain acoustic features corresponding to the target text, wherein the acoustic model includes a speech encoding module, and the speech encoding module is used to generate a speech phoneme vector based on a text phoneme vector; a synthesis module, used to synthesize the speech corresponding to the target text based on the acoustic features.
[0012] A third aspect of an embodiment of the present application provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the electronic device, wherein the processor implements the steps of the speech synthesis method provided in the first aspect when executing the computer program.
[0013] A fourth aspect of an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the speech synthesis method provided in the first aspect are implemented.
[0014] The speech synthesis method, device, electronic device and storage medium provided in the embodiments of the present application have the following beneficial effects:
[0015] The present application obtains a text phoneme sequence corresponding to the target text, and then uses a trained acoustic model to perform feature conversion processing on the phoneme sequence to obtain a text phoneme vector, and generates a corresponding speech phoneme vector based on the text phoneme vector, and generates the acoustic features corresponding to the target text based on the text phoneme vector and the speech phoneme vector. Then, based on the acoustic features, the speech corresponding to the target text is synthesized. Based on the above method, by adding a speech coding module to the traditional acoustic model, the acoustic model has the ability to generate speech phoneme vectors based on text phoneme vectors. In the process of converting text to speech, it is possible to simultaneously use text phoneme vectors and speech phoneme vectors, thereby improving the rhythmic effect of the synthesized speech. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.
[0017] Figure 1 A flowchart of a speech synthesis method provided in an embodiment of the present application;
[0018] Figure 2 A flow chart of reasoning of an acoustic model provided in an embodiment of the present application;
[0019] Figure 3 A flowchart of a method for implementing a method of training an acoustic model in a speech synthesis method provided in an embodiment of the present application;
[0020] Figure 4 A schematic diagram of a training process of an acoustic model provided in an embodiment of the present application;
[0021] Figure 5 A flowchart of a method for determining that an acoustic model is trained to a convergence state in a speech synthesis method provided in an embodiment of the present application;
[0022] Figure 6 A flowchart of a method for obtaining a reference speech phoneme vector in a speech synthesis method provided in an embodiment of the present application;
[0023] Figure 7 A flowchart of a method for obtaining a text phoneme sequence in a speech synthesis method provided in an embodiment of the present application;
[0024] Figure 8 A basic structural block diagram of a speech synthesis device provided in an embodiment of the present application;
[0025] Fig. 9A basic structural block diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0026] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.
[0027] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, wholes, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or combinations thereof.
[0028] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.
[0029] As used in the specification and appended claims of this application, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting", depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce [described condition or event] is detected" or "in response to detecting [described condition or event]", depending on the context.
[0030] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.
[0031] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the statements "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. that appear in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized in other ways. "Multiple" means "two or more".
[0032] In order to make the objectives, technical solutions and advantages of the present application more clear and understandable, the present application will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0033] Please refer to Figure 1 , Figure 1 which is a flowchart of the implementation of a speech synthesis method provided by an embodiment of the present application. Specifically, it may include the following steps S11 to S13.
[0034] S11: Obtain a text phoneme sequence corresponding to the target text, where the target text is the text to be converted into speech.
[0035] In this embodiment, the phoneme sequence is a structure composed of multiple phonemes. A phoneme is a concept in phonetics and is the smallest speech unit divided according to the natural attributes of speech, used to describe the pronunciation in a language. Different languages correspond to different phoneme tables. In this embodiment, the target text can be automatically generated by an intelligent device or input by an external user and needs to be converted into speech, and the language type of the target text is not limited. In this embodiment, for the target text, it can be analyzed based on the pronunciation actions of the words and phrases in the syllable. One action constitutes one phoneme. Taking Chinese as an example, each Chinese character contains one or more phonemes. For example, the Chinese character "啊(ā)" has only one phoneme, the Chinese character "爱(ài)" has two phonemes, and the Chinese character "代(dài)" has three phonemes, etc. In this embodiment, by performing phoneme recognition on each word and phrase of the target text one by one, the phoneme sequence corresponding to each word and phrase can be determined. Finally, according to the position order of each word and phrase in the target text, the phoneme sequences corresponding to all words and phrases are spliced together in the position order to form the text phoneme sequence corresponding to the target text.
[0036] S12: Perform feature conversion processing on the phoneme sequence by using a trained acoustic model to obtain a text phoneme vector, generate a corresponding speech phoneme vector according to the text phoneme vector, and generate an acoustic feature corresponding to the target text according to the text phoneme vector and the speech phoneme vector.
[0037] In this embodiment, a speech coding module is added to the trained acoustic model, that is, the acoustic model includes a text coding module, a speech coding module, a variable-length prediction module and a decoding module. The text coding module is used to obtain a text phoneme vector, which is represented by a matrix vector, and the matrix vector contains multiple elements, each of which contains text information of multiple dimensions. Wherein, the number of dimensions of each element is determined by the network structure setting of the text coding module, for example, it can be 256 dimensions or 512 dimensions. Each element in the matrix vector represents a phoneme. The speech coding module is used to obtain the corresponding speech phoneme vector according to the text phoneme vector, and the speech phoneme vector is also represented by a matrix vector, and the matrix vector contains multiple elements, and each element contains sound information of multiple dimensions. Wherein, the number of dimensions of each element is determined by the network structure setting of the speech coding module, for example, it can be 256 dimensions or 512 dimensions. It can be understood that the text phoneme vector has the same number of elements as the speech phoneme vector, and each element represents a phoneme. The number of dimensions of the elements can be different. For example, the number of dimensions of each element in the text phoneme vector is 256, and the number of dimensions of the elements in the speech phoneme vector can be 256 or 512. The variable length prediction module is used to predict the duration of the vector obtained by concatenating the text phoneme vector and the speech phoneme vector, and expand the text phoneme vector and the speech phoneme vector to the frame level vector. The decoding module is used to convert the frame level vector into acoustic features. For example, please refer to Figure 2 , Figure 2 The following is a flowchart of an acoustic model inference provided in an embodiment of the present application. Figure 2 As shown, when using the trained acoustic model, after the text phoneme sequence of the target text is input into the acoustic model, the text phoneme sequence of the target text is first preprocessed to obtain a corresponding phoneme vector, which is represented as a matrix vector, and each element in the matrix vector records the information of its corresponding phoneme itself. By inputting the matrix vector into the text encoding module for feature conversion processing, a text phoneme vector can be obtained, and each element in the text phoneme vector records the text information of its corresponding phoneme. By inputting the text phoneme vector into the speech encoding module for feature conversion processing, a speech phoneme vector can be obtained, and each element in the speech phoneme vector records the sound information of its corresponding phoneme. By splicing the obtained text phoneme vector and the speech phoneme vector, the spliced vector is input into the variable length prediction module for prediction to obtain a frame level vector. Finally, the frame level vector is converted through the decoding module to obtain the acoustic features corresponding to the target text. In this embodiment, the concatenation of the text phoneme vector and the speech phoneme vector is specifically to concatenate the elements in the matrix vector one by one, and obtain a concatenated matrix vector as the input of the variable-length prediction module.
[0038] S13: Synthesizing speech corresponding to the target text according to the acoustic features.
[0039] In this embodiment, the speech corresponding to the target text can be obtained by inputting the acoustic features generated by the acoustic model into the vocoder for speech synthesis. In this embodiment, the main function of the vocoder is to convert the acoustic features into a playable speech waveform.
[0040] From the above, it can be seen that the speech synthesis method provided in the embodiment of the present application obtains the phoneme sequence corresponding to the target text to be converted into speech, and uses an acoustic model with a speech coding module to perform feature conversion processing on the text phoneme sequence. The speech coding module in the acoustic model can be used to generate speech phoneme vectors according to text phoneme vectors, thereby obtaining acoustic features that contain both text phoneme vectors and speech phoneme vectors. Then, the speech corresponding to the target text is synthesized based on the acoustic features. Based on the above method, by adding a speech coding module to the traditional acoustic model, the acoustic model has the ability to generate speech phoneme vectors according to text phoneme vectors, thereby achieving the simultaneous use of text phoneme vectors and speech phoneme vectors in the process of converting text to speech, thereby improving the rhythmic effect of the synthesized speech.
[0041] In some embodiments of the present application, an acoustic model is specifically built in combination with a speech coding module, and the speech coding module is jointly trained when the acoustic model is trained using training sample data to obtain a trained acoustic model, which is used to generate acoustic features according to a text phoneme sequence. In this embodiment, the acoustic model includes a text coding module, a speech coding module, a variable-length prediction module, and a decoding module, wherein the text coding module, the speech coding module, and the decoding module are built through a Transformer structure, and the Transformer structure is a non-autoregressive speech synthesis structure, and the text coding module, the speech coding module, and the decoding module use self-attention and 1D convolution structures, which can make full use of context information for prediction of speech features, and the non-autoregressive method can improve the efficiency of acoustic model training and reasoning. Moreover, a speech coding module is added during the construction process, so that the acoustic model is combined with the speech coding module for joint training, and in the acoustic model, the speech coding module takes the text phoneme vector as input and the speech phoneme vector as output, and the speech coding module is trained to generate the corresponding speech phoneme vector according to the input text phoneme vector. The acoustic model is trained to obtain a text phoneme vector and a speech phoneme vector according to an input text phoneme sequence, and then simultaneously uses the obtained text phoneme vector and speech phoneme vector to generate acoustic features, and performs speech synthesis based on the acoustic features to improve the rhythmic effect of the synthesized speech. It should be noted that, in this embodiment, the pre-selected trained acoustic model has the ability to generate acoustic features of text according to the text phoneme sequence of the text.
[0042] In some embodiments of this application, please refer to Figure 3 , Figure 3 A flowchart of a method for training an acoustic model in a speech synthesis method provided in an embodiment of the present application. Figure 3 As shown, it may specifically include steps S31 to S34.
[0043] S31: Building an acoustic model, wherein the acoustic model includes a text encoding module, a speech encoding module, a variable length prediction module, and a decoding module;
[0044] S32: constructing training sample data for training the acoustic model, wherein the training sample data includes a text phoneme sequence, a speech phoneme sequence and a standard acoustic feature of a training text, wherein the speech phoneme sequence is used to obtain a standard speech phoneme vector of a training sample;
[0045] S33: inputting the text phoneme sequence in the training sample into the text encoding module to obtain a predicted text phoneme vector; inputting the text phoneme vector into the speech encoding module to obtain a predicted speech phoneme vector; merging the predicted text phoneme vector and the predicted speech phoneme vector and inputting them into the variable-length prediction module to obtain a frame-level vector; inputting the frame-level vector into the decoding module to obtain a predicted acoustic feature;
[0046] S34: Update the parameters of the speech encoding module according to the predicted speech phoneme vector and the standard speech phoneme vector, and update the parameters of the decoding module according to the predicted acoustic features and the standard acoustic features, until the acoustic model is trained to a convergence state, so as to obtain a trained acoustic model.
[0047] In this embodiment, by combining the speech coding module to build an acoustic model, the acoustic model can include a text coding module, a speech coding module, a variable length prediction module and a decoding module. By using training sample data to jointly train the above text coding module, speech coding module, variable length prediction module and decoding module, the acoustic model has the ability to generate acoustic features according to the text phoneme sequence. For example, please refer to Figure 4 , Figure 4 A schematic diagram of the training process of an acoustic model provided in an embodiment of the present application. Figure 4As shown, the text encoding module in the acoustic model is trained to obtain a predicted text phoneme vector based on a text phoneme sequence, the speech encoding module is trained to generate a speech phoneme vector based on a text phoneme vector, the variable-length prediction module is trained to expand the text phoneme vector and the speech phoneme vector into a frame-level vector, and the decoding module is trained to convert the frame-level vector into an acoustic feature. In this embodiment, a number of training sample data can be pre-constructed, each of which includes a text phoneme sequence, a speech phoneme sequence, and a standard acoustic feature of a training text. In the process of training the acoustic model, the speech phoneme sequence is used to obtain a standard speech phoneme vector of the training sample, and the parameters of the speech encoding module are modified using the standard speech phoneme vector as a benchmark. In this embodiment, when training the acoustic model, specifically, the text phoneme sequence in the training sample is input into the text encoding module to obtain a predicted text phoneme vector; the text phoneme vector is input into the speech encoding module to obtain a predicted speech phoneme vector; the predicted text phoneme vector and the predicted speech phoneme vector are combined and input into the variable length prediction module to obtain a frame level vector; the frame level vector is input into the decoding module to obtain a predicted acoustic feature. In the above process, the parameters of the speech encoding module are updated according to the predicted speech phoneme vector output by the speech encoding module and the standard speech phoneme vector obtained by the speech phoneme sequence in the training sample, and the parameters of the decoding module are updated according to the predicted acoustic features output by the decoding module and the standard acoustic features in the training sample, until the acoustic model is trained to a convergence state, at which time a trained acoustic model can be obtained.
[0048] In some specific embodiments, the variable length prediction module obtains the phoneme duration by forcibly aligning the text phoneme vector and the speech phoneme vector. For example, the frame length of the phoneme eng2 is 12.5ms, and it is assumed here that it lasts for 100ms, that is, the phoneme duration frame length is 100 / 12.5=8. When using the acoustic model, the input of the variable length prediction module is the text phoneme vector and the speech phoneme vector, and the output is the number of frames corresponding to each phoneme. For example, in the above example, the phoneme eng2 corresponds to a duration frame length of 8 frames, and the frame level vector is obtained by expanding by copying 8 copies of the vector corresponding to the phoneme.
[0049] In some embodiments of this application, please refer to Figure 5 , Figure 5 A flowchart of a method for determining that the acoustic model is trained to a convergence state in the speech synthesis method provided in the embodiment of the present application. Figure 5 As shown, it may specifically include steps S51 to S53.
[0050] S51: Obtaining a loss function of the speech coding module and a loss function of the acoustic model;
[0051] S52: using the loss function of the speech encoding module to calculate a first loss value between the predicted speech phoneme vector and the standard speech phoneme vector, and judging whether the first loss value satisfies a preset first threshold condition, and using the loss function of the acoustic model to calculate a second loss value between the predicted acoustic feature and the standard acoustic feature, and judging whether the second loss value satisfies a preset second threshold condition;
[0052] S53: If the first loss value satisfies a preset first threshold condition and the second loss value satisfies a preset second threshold condition, it is determined that the acoustic model is trained to a convergence state.
[0053] In this embodiment, joint training is performed based on the addition of a speech coding module to the acoustic model. When training the acoustic model, the loss function of the speech coding module and the loss function of the acoustic model can be configured, and the loss function of the speech coding module and the loss function of the acoustic model are combined to determine whether the acoustic model has been trained to a convergence state. The output of the speech coding module is a predicted speech phoneme vector. The loss function of the speech coding module can be used to calculate the first loss value between the predicted speech phoneme vector and the standard speech phoneme vector, and to determine whether the first loss value meets the preset first threshold condition, thereby determining whether the speech coding module is trained to a convergence state. The output of the acoustic model is a predicted acoustic feature. The loss function of the acoustic model can be used to calculate the second loss value between the predicted acoustic feature and the standard acoustic feature, and to determine whether the second loss value meets the preset second threshold condition, thereby determining whether the acoustic model is trained to a convergence state. When the first loss value meets the preset first threshold condition and the second loss value meets the preset second threshold condition, it is determined that the acoustic model is trained to a convergence state.
[0054] In some embodiments of this application, please refer to Figure 6 , Figure 6 A flowchart of a method for obtaining a reference speech phoneme vector in the speech synthesis method provided in an embodiment of the present application. Figure 6 As shown, it may specifically include steps S61 to S63.
[0055] S61: Performing speech recognition on the speech phoneme sequence of the training sample using an unsupervised speech recognition model to obtain a plurality of speech latent vectors corresponding to each phoneme in the speech phoneme sequence;
[0056] S62: For each phoneme, taking the mean of the multiple speech latent vectors as the vector representation of the phoneme;
[0057] S63: concatenate the vector representations corresponding to all the phonemes in the speech phoneme sequence to obtain a standard speech phoneme vector of the training sample.
[0058] In this embodiment, the unsupervised speech recognition model uses a speech phoneme sequence to be trained in a self-supervised manner to obtain a vector representation of the speech. Exemplarily, the unsupervised speech recognition model is constructed by a wav2vec convolutional neural network model. In this embodiment, the vector representation of the speech is a speech latent vector extracted by the wav2vec convolutional neural network model. The wav2vec convolutional neural network model includes multiple network layers. The speech latent vector of the network layer with the highest phoneme accuracy is determined according to the phoneme accuracy corresponding to the speech latent vectors of different network layers in the wav2vec convolutional neural network model. In the wav2vec convolutional neural network model, the speech latent vector is set as a vector of one frame. In this embodiment, for the speech phoneme sequence of the training sample, multiple speech latent vectors corresponding to each phoneme in the speech phoneme sequence can be obtained according to the duration and number of frames of each phoneme in the speech phoneme sequence. For each phoneme, the mean of the multiple speech latent vectors corresponding to the phoneme is calculated, and the mean of the multiple speech latent vectors is used as the vector representation of the phoneme. After the vector representations corresponding to all the phonemes in the speech phoneme sequence are obtained in the above manner, the vector representations corresponding to all the phonemes in the speech phoneme sequence can be concatenated to obtain the standard speech phoneme vector of the training sample. It should be noted that the duration and number of frames of each phoneme in the speech phoneme sequence can be obtained by forcibly aligning the text phoneme sequence and the speech phoneme sequence in the training sample.
[0059] In some embodiments of this application, please refer to Figure 7 , Figure 7 A flowchart of a method for obtaining a text phoneme sequence in the speech synthesis method provided in an embodiment of the present application. Figure 7 As shown, it may specifically include steps S71 to S75.
[0060] S71: Obtaining a target text to be converted into speech;
[0061] S72: performing word splitting processing on the target text to obtain a plurality of words;
[0062] S73: for each word, determining the language type of the word, and acquiring a phoneme table matching the word according to the language type;
[0063] S74: Acquire a text phoneme sequence corresponding to the word according to the phoneme table;
[0064] S75: Concatenate the text phoneme sequences corresponding to all the words according to the position order of the words in the target text to obtain the text phoneme sequence corresponding to the target text.
[0065] In this embodiment, the target text may be a text automatically generated by an intelligent device such as a computer, or a text input by a user through human-computer interaction. The target text may be a text containing only one language, or a mixed text containing multiple languages. In this embodiment, after obtaining the target text, the target text may be subjected to word segmentation processing according to the minimum granularity of text segmentation, and the target text may be segmented into multiple words. It is understandable that for Chinese text, the minimum granularity of text segmentation is a character, and for English text, the minimum granularity of text segmentation is a word. In this embodiment, the phoneme tables corresponding to different language types are different. For each word obtained by the segmentation, the word may first be subjected to language type identification to determine the language type of the word, and the phoneme table matching the word may be obtained according to the language type of the word. Furthermore, the text phoneme sequence corresponding to the word may be determined according to the phoneme table matching the word. When all the words in the target text have obtained the corresponding text phoneme sequence, the text phoneme sequences corresponding to all the words may be spliced according to the position order of the words in the target text, and the text phoneme sequence corresponding to the target text may be obtained. In this embodiment, the recognition of language type can be achieved by pre-training a deep convolutional neural network model for language type recognition, and then determining the language type of the word through the deep convolutional neural network model. Through this embodiment, the speech synthesis method can perform speech synthesis for texts of different language types, which has strong applicability and can be applied to different language scenarios.
[0066] In some embodiments of the present application, after the multiple words are obtained after the word splitting process, the punctuation marks in the target text can be filtered out through data cleaning process to avoid interference with the subsequent speech synthesis process.
[0067] It is understandable that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.
[0068] In some embodiments of this application, please refer to Figure 8 , Figure 8 This is a basic structural block diagram of a speech synthesis device provided in an embodiment of the present application. The various units included in the device in this embodiment are used to perform the various steps in the above method embodiment. For details, please refer to the relevant description in the above method embodiment. For ease of explanation, only the parts related to this embodiment are shown. Figure 8As shown, the speech synthesis device includes: an acquisition module 81, a conversion module 82 and a synthesis module 83. Among them: the acquisition module 81 is used to obtain the text phoneme sequence corresponding to the target text, and the target text is the text to be converted into speech. The conversion module 82 is used to use the trained acoustic model to perform feature conversion processing on the phoneme sequence to obtain a text phoneme vector, and generate a corresponding speech phoneme vector based on the text phoneme vector, and generate the acoustic features corresponding to the target text based on the text phoneme vector and the speech phoneme vector. The synthesis module 83 is used to synthesize the speech corresponding to the target text according to the acoustic features.
[0069] It should be understood that the above-mentioned speech synthesis device corresponds one-to-one to the above-mentioned speech synthesis method, and will not be described in detail here.
[0070] In some embodiments of this application, please refer to Fig. 9 , Fig. 9 This is a basic structural block diagram of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 9 of this embodiment includes: a processor 91, a memory 92, and a computer program 93 stored in the memory 92 and executable on the processor 91, such as a program of a speech synthesis method. When the processor 91 executes the computer program 93, the steps in each embodiment of the above-mentioned speech synthesis method are implemented. Alternatively, when the processor 91 executes the computer program 93, the functions of each module in the embodiment corresponding to the above-mentioned speech synthesis device are implemented. Please refer to the relevant description in the embodiment for details, which will not be repeated here.
[0071] Exemplarily, the computer program 93 may be divided into one or more modules (units), which are stored in the memory 92 and executed by the processor 91 to complete the present application. The one or more modules may be a series of computer program instruction segments capable of completing specific functions, which are used to describe the execution process of the computer program 93 in the electronic device 9. For example, the computer program 93 may be divided into an acquisition module, a conversion module, and a synthesis module, and the specific functions of each module are as described above.
[0072] The electronic device may include, but is not limited to, a processor 91 and a memory 92. Those skilled in the art will appreciate that Fig. 9 It is only an example of the electronic device 9 and does not constitute a limitation of the electronic device 9. It may include more or fewer components than shown in the figure, or a combination of certain components, or different components. For example, the electronic device may also include input and output devices, network access devices, buses, etc.
[0073] The processor 91 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc.
[0074] The memory 92 may be an internal storage unit of the electronic device 9, such as a hard disk or memory of the electronic device 9. The memory 92 may also be an external storage device of the electronic device 9, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. equipped on the electronic device 9. Further, the memory 92 may also include both an internal storage unit of the electronic device 9 and an external storage device. The memory 92 is used to store the computer program and other programs and data required by the electronic device. The memory 92 may also be used to temporarily store data that has been output or is to be output.
[0075] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.
[0076] The embodiment of the present application further provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps in the above-mentioned various method embodiments can be implemented. In this embodiment, the computer-readable storage medium can be non-volatile or volatile.
[0077] An embodiment of the present application provides a computer program product. When the computer program product runs on a mobile terminal, the mobile terminal can implement the steps in the above-mentioned method embodiments when executing the computer program product.
[0078] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.
[0079] If the integrated module / unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, and can also be completed by instructing the relevant hardware through a computer program. The computer program can be stored in a computer-readable storage medium, and the computer program can implement the steps of the above-mentioned various method embodiments when executed by the processor. Among them, the computer program includes computer program code, and the computer program code can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, disk, optical disk, computer memory, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), electric carrier signal, telecommunication signal and software distribution medium. It should be noted that the content contained in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electric carrier signals and telecommunication signals.
[0080] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.
[0081] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.
Claims
1. A speech synthesis method, characterized in that: include: Obtaining a text phoneme sequence corresponding to a target text, wherein the target text is a text to be converted into speech; Using a trained acoustic model to perform feature conversion processing on the text phoneme sequence to obtain a text phoneme vector, and generating a corresponding speech phoneme vector based on the text phoneme vector, and generating an acoustic feature corresponding to the target text based on the text phoneme vector and the speech phoneme vector; The speech corresponding to the target text is synthesized according to the acoustic features.
2. The speech synthesis method according to claim 1, characterized in that: Before the step of using the trained acoustic model to perform feature conversion processing on the text phoneme sequence, the method further includes: An acoustic model is built in combination with a speech coding module. When the acoustic model is trained using training sample data, the speech coding module is jointly trained to obtain a trained acoustic model, which is used to generate acoustic features based on a text phoneme sequence.
3. The speech synthesis method according to claim 2, characterized in that: The step of building an acoustic model in combination with a speech coding module, and jointly training the speech coding module when training the acoustic model with training sample data to obtain a trained acoustic model includes: Building an acoustic model, wherein the acoustic model includes a text encoding module, a speech encoding module, a variable length prediction module, and a decoding module; Constructing training sample data for training the acoustic model, the training sample data including a text phoneme sequence, a speech phoneme sequence and standard acoustic features of a training text, the speech phoneme sequence being used to obtain a standard speech phoneme vector of the training sample; Inputting the text phoneme sequence in the training sample into the text encoding module to obtain a predicted text phoneme vector; inputting the text phoneme vector into the speech encoding module to obtain a predicted speech phoneme vector; merging the predicted text phoneme vector and the predicted speech phoneme vector and inputting them into the variable length prediction module to obtain a frame level vector; inputting the frame level vector into the decoding module to obtain a predicted acoustic feature; The parameters of the speech encoding module are updated according to the predicted speech phoneme vector and the standard speech phoneme vector, and the parameters of the decoding module are updated according to the predicted acoustic features and the standard acoustic features, until the acoustic model is trained to a convergence state, so as to obtain a trained acoustic model.
4. The speech synthesis method according to claim 3, characterized in that: The step of updating the parameters of the speech encoding module according to the predicted speech phoneme vector and the standard speech phoneme vector and updating the parameters of the decoding module according to the predicted acoustic features and the standard acoustic features until the acoustic model is trained to a convergence state comprises: Obtaining a loss function of the speech coding module and a loss function of the acoustic model; The first loss value between the predicted speech phoneme vector and the standard speech phoneme vector is calculated by using the loss function of the speech encoding module, and whether the first loss value satisfies a preset first threshold condition is determined; and the second loss value between the predicted acoustic feature and the standard acoustic feature is calculated by using the loss function of the acoustic model, and whether the second loss value satisfies a preset second threshold condition is determined; If the first loss value satisfies a preset first threshold condition and the second loss value satisfies a preset second threshold condition, it is determined that the acoustic model is trained to a convergence state.
5. The speech synthesis method according to any one of claims 3 or 4, characterized in that: The step of obtaining the standard speech phoneme vector of the training sample comprises: Using an unsupervised speech recognition model to perform speech recognition on the speech phoneme sequence of the training sample to obtain a plurality of speech latent vectors corresponding to each phoneme in the speech phoneme sequence; For each phoneme, taking the mean of the multiple speech latent vectors as the vector representation of the phoneme; The vector representations corresponding to all the phonemes in the speech phoneme sequence are concatenated to obtain a standard speech phoneme vector of the training sample.
6. The speech synthesis method according to claim 1, characterized in that: The step of obtaining a text phoneme sequence corresponding to the target text comprises: Obtain the target text to be converted into speech; Performing word splitting processing on the target text to obtain multiple words; For each word, determine the language type of the word, and obtain a phoneme table matching the word according to the language type; According to the phoneme table, obtaining a text phoneme sequence corresponding to the word; According to the position order of the words in the target text, the text phoneme sequences corresponding to all the words are concatenated to obtain the text phoneme sequence corresponding to the target text.
7. The speech synthesis method according to claim 6, characterized in that: After the step of performing word splitting processing on the target text to obtain a plurality of words, the method further includes: Perform data cleaning processing on the multiple words.
8. A speech synthesis device, characterized in that: include: An acquisition module, used to acquire a text phoneme sequence corresponding to a target text, wherein the target text is a text to be converted into speech; A conversion module, used to perform feature conversion processing on the phoneme sequence using a trained acoustic model to obtain acoustic features corresponding to the target text, wherein the acoustic model includes a speech encoding module, and the speech encoding module is used to generate a speech phoneme vector according to the text phoneme vector; A synthesis module is used to synthesize the speech corresponding to the target text according to the acoustic features.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.