A speech synthesis model training method, device, electronic equipment and medium
By acquiring and processing source audio data and target speech style features, and updating the parameters of the speech synthesis model, the problem of insufficient generalization ability of the speech synthesis model in the existing technology is solved, the speech style similarity is improved, and more realistic speech synthesis is achieved Effect.
Patent Information
- Application Number
- CN202110397569.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2021-04-13
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2041-04-13
AI Technical Summary
The existing speech synthesis model has limited generalization ability during speech style transfer, resulting in a low similarity in speech styles of synthetic audio data.
By obtaining the source audio data and the target speech style characteristics, calling the speech conversion model and the initial speech synthesis model for processing, the first predicted audio data and the second predicted audio data are generated, and the parameters of the initial speech synthesis model are updated based on these data to improve the The generalization ability and speech style similarity of the model.
The generalization ability of the speech synthesis model is enhanced, and the speech style similarity of the synthetic audio data is improved, making the generated audio data closer to the target speech style.
Smart Images

Figure CN115206281B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of speech technology, and in particular to a speech synthesis model training method, device, electronic equipment and medium. Background Art
[0002] With the rapid development of speech technology, speech synthesis technology (TTS) has gradually been used more and more in life. Speech style transfer is an important field in the field of speech synthesis technology. Speech style transfer refers to assigning the speech style of a certain speaker to the synthesized audio data. At present, the speech synthesis model training for speech style transfer is to train an average model based on multi-person speech corpus, and then fine-tune the target speaker. However, in practice, it is found that the generalization ability of the model trained in this way is limited. When the model synthesizes audio data according to this training method, the synthesized audio data has a low speech style similarity. Therefore, how to improve the speech style similarity in the process of speech style transfer is an urgent problem to be solved. Summary of the invention
[0003] The embodiments of the present application provide a speech synthesis model training method, device, electronic device and medium, which help to enhance the generalization ability of the speech synthesis model and improve the speech style similarity of synthesized audio data.
[0004] On the one hand, an embodiment of the present application provides a method for training a speech synthesis model, the method comprising:
[0005] Acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data;
[0006] Calling a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0007] Extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0008] The parameters in the initial speech synthesis model are updated based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0009] On the other hand, an embodiment of the present application provides a speech synthesis model training device, the device comprising:
[0010] An acquisition module, used to acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data;
[0011] A processing module, configured to call a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0012] The processing module is further used to extract source text data from the source audio data, and call the initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0013] The processing module is also used to update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0014] On the other hand, an embodiment of the present application provides an electronic device, characterized in that the electronic device includes a processor and a storage device, the processor and the storage device are connected to each other, wherein the storage device is used to store computer program instructions, and the processor is configured to perform the following steps:
[0015] Acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data;
[0016] Calling a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0017] Extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0018] The parameters in the initial speech synthesis model are updated based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0019] In another aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the computer program instructions are used to perform the following steps:
[0020] Acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data;
[0021] Calling a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0022] Extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0023] The parameters in the initial speech synthesis model are updated based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0024] The embodiment of the present application can obtain source audio data and target voice style features, call the voice conversion model to process the source audio data and the target voice style features, obtain first predicted audio data, thereby obtaining audio data corresponding to the target voice style features, call the initial voice synthesis model to process the source text data and the target voice style features, obtain second predicted audio data, and then update the parameters in the initial voice synthesis model based on the first predicted audio data and the second predicted audio data to train the initial voice synthesis model. This can help enhance the generalization ability of the trained voice synthesis model and improve the voice style similarity of the synthesized audio data. BRIEF DESCRIPTION OF THE DRAWINGS
[0025] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0026] Figure 1 It is a flowchart of a speech synthesis model training method provided in an embodiment of the present application;
[0027] Figure 2 is a structural diagram of a speech conversion model provided in an embodiment of the present application;
[0028] Figure 3 is a structural diagram of a speech conversion model provided in an embodiment of the present application;
[0029] Figure 4 It is a flowchart of a speech conversion model training method provided in an embodiment of the present application;
[0030] Figure 5 It is a flowchart of a method for obtaining audio data provided in an embodiment of the present application;
[0031] Figure 6 It is a flowchart of a method for obtaining audio data provided in an embodiment of the present application;
[0032] Figure 7It is a flowchart of a speech synthesis model training method provided in an embodiment of the present application;
[0033] Figure 8 It is a flowchart of a method for obtaining audio data provided in an embodiment of the present application;
[0034] Fig. 9 It is a flowchart of a method for obtaining audio data provided in an embodiment of the present application;
[0035] Fig.10 It is a structural schematic diagram of a speech synthesis model training device provided in an embodiment of the present application;
[0036] Fig.11 It is a structural schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0037] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0038] The embodiment of the present application proposes a speech synthesis model training scheme, which can obtain source audio data and target speech style features, call a speech conversion model to process the source audio data and the target speech style features, obtain first predicted audio data, thereby obtaining audio data corresponding to the target speech style features, call an initial speech synthesis model to process the source text data and the target speech style features, obtain second predicted audio data, and then update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model. This can help enhance the generalization ability of the trained speech synthesis model and improve the speech style similarity of the synthesized audio data.
[0039] In the present application, audio data refers to digitized sound data, such as source audio data, target audio data, sample audio data, first predicted audio data or second predicted audio data, etc. The acoustic spectrum (Acoustic Spectrum) in the present application is a tool used to describe the components contained in the sound and the distribution pattern of acoustic energy in the timbre, such as the first predicted acoustic spectrum or the second predicted acoustic spectrum, etc. The acoustic spectrum in the present application can be a tool such as a Mel spectrum, a spectrogram, a Mel cepstrum, etc., which is not limited here. The essence of the acoustic spectrum such as the Mel spectrum and the spectrogram is a frequency spectrum sequence. In other words, the acoustic spectrum such as the Mel spectrum and the spectrogram can be composed of multiple frames of frequency domain data.
[0040] In the present application, speech style may refer to the characteristics of timbre, accent, basic pitch, etc. in audio data, which are not limited here. In the present application, speech style is represented by speech style features, such as target speech style features may represent features corresponding to speech style features of target audio data, and sample speech style features may represent features corresponding to speech style features of sample audio data. Optionally, the speech style features may use vectors or matrices to represent features of the corresponding audio data, or in other words, the speech style features may use a set of vectors or a matrix to represent the identifier of the object corresponding to the audio data, and the identifier is used to indicate the speech style features of the audio data of the object.
[0041] In this application, rhythm is used to describe the relative pitch and interval time between syllables in audio data, which is not limited here. In this application, rhythm is represented by rhythm features. For example, the source rhythm feature in this application can represent the feature corresponding to the rhythm in the source audio data, and the sample rhythm feature can represent the feature corresponding to the rhythm in the sample audio data. Optionally, the rhythm feature can usually be represented by a vector or a matrix, which is not limited here.
[0042] The technical solution of the present application can be applied to electronic devices, and the technical solution of the present application can be applied to electronic devices, which can be terminals or servers, and the present application does not limit it. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The terminal can be a smart phone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to this.
[0043] The technical solution of this application can be applied to the field of speech data technology. The key technologies of speech technology (Speech Technology) include automatic speech recognition technology (ASR), speech synthesis technology (TTS) and voiceprint recognition technology. Enabling computers to listen, see, speak and feel is the development direction of human-computer interaction in the future, among which speech has become one of the most promising human-computer interaction methods in the future.
[0044] The technical solution of the present application can also be applied to mathematical intelligence education. For example, one or more virtual teachers can be trained through the model training method provided by the present application, so that the one or more virtual teachers have the ability to speak fluently and even have the ability to speak in multiple languages. Specifically, multilingual audio data with target style characteristics can be trained through the speech conversion model of the present solution, so that the one or more virtual teachers can have the ability to speak in multiple languages. It can be seen that the technical solution of the present application can also be applied in other fields involving speech synthesis, such as smart speakers, smart homes, audio book software, virtual anchors, virtual tour guides, virtual teaching assistants, intelligent customer service, etc.
[0045] In a possible implementation, the technical solution of the present application can also be used for training a cross-language speech synthesis model, so that a cross-language speech synthesis model can be obtained. Specifically, the source audio data of the target language and the target speech style features can be processed by calling the speech conversion model to obtain the first predicted audio data of the target language, and then the initial speech synthesis model is trained by the first predicted audio data, the source text data and the target speech style features. For example, when the speaker can only speak Chinese, but it is necessary to generate English audio data corresponding to the speech style of the speaker, the source audio data with English text content can be obtained, and the speech conversion model is called to process the target speech style features and the source audio data, so that the first predicted audio data with English text content can be obtained, so that after training the initial speech synthesis model, the English audio data corresponding to the target speech style can be obtained. Similarly, a speech synthesis model that can synthesize more target language types can be obtained according to the adjustment of the type of the target language, such as a Cantonese model, a Sichuanese model or a Northeastern dialect model, etc., which greatly enriches the language ability of the speaker.
[0046] In a possible implementation, the technical solution of the present application can also be applied to change one voice style into the voice style of another speaking object. Specifically, the text data features corresponding to the audio data of one voice style and the target voice style features of another voice style can be obtained, and the voice conversion model is called to process the text data features and the target voice style features to obtain the target predicted audio data. The voice style of the target predicted audio data is another voice style, and the audio content is the content corresponding to the audio data of the one voice style, that is, it is possible to change one voice style into the voice style of another speaking object.
[0047] It is understood that the above scenarios are only examples and do not constitute a limitation on the application scenarios of the technical solutions provided in the embodiments of the present application. The technical solutions of the present application can also be applied to other scenarios. For example, it is known to those skilled in the art that with the evolution of the system architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0048] Based on the above description, the present application embodiment proposes a speech synthesis model training method. Figure 1 , Figure 1 It is a flowchart of a speech synthesis model training method provided in an embodiment of the present application. The speech synthesis model training method may include steps S101-S104.
[0049] S101, obtaining source audio data and target speech style features.
[0050] The source audio data may be obtained from various audio databases, or may be obtained by collecting audio data of various objects to obtain an audio data set, and the source audio data may be determined from the audio data set. The target speech style feature may represent a feature corresponding to the speech style feature of the target audio data, and the target speech style feature may be represented by a vector or a matrix.
[0051] In a possible implementation, the target speech style feature may input the target audio data of indefinite length into a pre-trained speaker recognition network, map the target audio data of indefinite length into a target vector, and the target vector may be of fixed length, so as to use the target vector as the target speech style feature. In some scenarios, the target vector may be referred to as an x-vector.
[0052] Optionally, after obtaining the target vector corresponding to the target audio data, the target vector may be processed, such as interpolating the target vector or modifying the value of the target vector, so as to create a new target vector and use the new target vector as the target speech style feature.
[0053] S102: Calling a speech conversion model to process source audio data and target speech style features to obtain first predicted audio data.
[0054] The speech conversion model can be used to convert the speech style of the source audio data into the speech style corresponding to the target speech style feature, thereby obtaining the first predicted audio data. In other words, the speech style of the first predicted audio data is the speech style corresponding to the target speech style feature, and the audio content of the first predicted audio data is the audio content indicated by the source audio data.
[0055] In a possible implementation, the speech conversion model may include a first encoding network and a first decoding network. The first encoding network is used to determine the source coding features of the source audio data, and the source coding features are feature vectors or feature matrices of the source audio data extracted by the first encoding network. The first encoder may be an expert system, which refers to an intelligent computer program system that contains a large amount of expert-level knowledge and experience in a certain field and can decouple the content of the source audio data. The first decoding network is used to receive the source coding features output by the first encoding network and obtain the target speech style features, thereby obtaining the first predicted audio data. The model of the first decoder sequence modeling is, such as a trained recurrent convolutional neural network (RNN) or a long short-term memory network (LSTM). For example, see Figure 2 , Figure 2 This is a structural diagram of a speech conversion model provided by an embodiment of the present application. In this diagram, a first encoding network in the speech conversion model receives source audio data, generates source coding features based on the source audio data, and then uses the source coding features and target speech style features as inputs to a first decoding network to obtain first predicted audio data.
[0056] In a possible implementation, calling a speech conversion model to process the source audio data and the target speech style features to obtain the first predicted audio data may specifically include the following steps: obtaining the source rhythm features in the source audio data, and determining the source coding features of the source audio data through a first encoding network; processing the source coding features, source rhythm features and target speech style features through a first decoding network to obtain the first predicted spectrum of the source audio data under the speech style of the target audio data; and constructing the first predicted audio data based on the first predicted spectrum. Among them, the source rhythm features are features corresponding to the rhythm in the source audio data. The rhythm in the first predicted audio data is the rhythm features corresponding to the source rhythm features, and the speech style is the speech style corresponding to the target speech style features. The first predicted spectrum may refer to the spectrum corresponding to the first predicted audio data, such as a Mel spectrum, a spectrogram (spectrogram), a Mel cepstrum, etc. When constructing the first predicted audio data based on the first predicted spectrum, the first predicted spectrum may be input into a neural network synthesizer to obtain the first predicted audio data. The neural network synthesizer is a neural network model trained with training samples, and can generate audio data based on the spectrum. For example, see Figure 3 , Figure 3: is a structural schematic diagram of a speech conversion model provided by an embodiment of the present application. In the figure, the first encoding network of the speech conversion model receives source audio data, generates source coding features according to the source audio data, and then uses the source coding features, source rhythm features and target speech style features as inputs to the first decoding network to obtain first predicted audio data. Inputting the source rhythm features into the first decoding network can make the sound in the generated first predicted audio data more stable, thereby improving the speech quality of the generated first predicted audio data.
[0057] In a possible implementation, before determining the source coding features of the source audio data through the first coding network, the following steps may also be performed: obtaining sample audio data, sample rhythm features and sample voice style features, and determining the sample coding features of the sample audio data through the first initial coding network; processing the sample coding features, sample rhythm features and sample voice style features through the first initial decoding network to obtain sample predicted audio data; updating the parameters in the first initial coding network and the first initial decoding network based on the sample audio data and the sample predicted audio data; if the first initial coding network after the parameter update and the first initial decoding network after the parameter update meet the first preset condition, then determining the first initial coding network after the parameter update as the first coding network, and determining the first initial decoding network after the parameter update as the first decoding network. Wherein, the sample rhythm feature is the feature corresponding to the rhythm in the sample audio data, and the sample voice style feature is the feature corresponding to the voice style of the sample audio data. The first initial coding network is used to determine the sample coding features of the sample audio data, and the sample coding features are the feature vectors or feature matrices of the sample audio data extracted by the first initial coding network. The first encoder may be an expert system, which refers to an intelligent computer program system that contains a large amount of knowledge and experience at the expert level in a certain field, and can decouple the content of the sample audio data. The first initial decoding network is used to receive the sample encoding features output by the first initial encoding network and obtain the sample rhythm features and the sample voice style features, thereby obtaining the sample prediction audio data. The first preset condition is used to indicate that the parameters and structures of the first initial encoding network and the first initial decoding network have reached the target satisfaction level, and the target satisfaction level can be used to indicate that the matching degree between the generated sample prediction audio data and the sample audio data meets the condition, such as determining the loss value between the sample prediction audio data and the sample audio data through the loss function, if the loss value satisfies the preset loss value condition, that is, the first initial encoding network and the first initial decoding network after the parameter update meet the first preset condition, the preset loss value condition can be that the loss value is less than the preset value, or it can be that the change between the loss values obtained after each update of the parameters of the first initial encoding network and the first initial decoding network is less than the preset value. Optionally, the sample measured audio data or the sample audio data can be represented by an audio signal, and the loss value between the sample predicted audio data and the sample audio data is determined by a loss function, which can also be said to be the loss value between the sample predicted audio data and the audio signal corresponding to the sample audio data. Among them, the preset loss value condition, preset value, etc. can be set by the model training user. The setting of the preset condition or preset value may be different for different models and scenarios, and is not limited here.
[0058] For example, see Figure 4 , Figure 4It is a flow chart of a speech conversion model training method provided by an embodiment of the present application. In the figure, it can be seen that after the sample audio data is input into the first initial encoding network, the sample encoding features corresponding to the sample audio data can be obtained. The first initial decoding network processes the sample encoding features, the sample rhythm features, and the sample voice style features to obtain the sample predicted audio data. The loss value between the sample predicted audio data and the sample audio data is determined by the loss function, thereby updating the parameter values of the first initial encoding network and the first initial decoding network. The loss function refers to the target to be minimized during the model training process, such as the mean square error loss function (MESloss), the mean absolute error loss function (MAE loss) or the cross entropy loss function, etc., which are not limited here. When the loss value between the sample predicted audio data and the sample audio data meets the preset loss value condition, the parameter update is stopped, and the updated first initial encoding network is determined as the first encoding network, and the first initial decoding network is determined as the first decoding network. The preset loss value condition can be that the loss value is less than the preset value, or the change between the loss values obtained after each update of the parameters of the first initial encoding network and the first initial decoding network is less than the preset value, which is not limited here. By training the first initial encoding network and the first initial decoding network with sample audio data, the obtained speech conversion model can construct predicted audio data according to speech style features.
[0059] Optionally, after obtaining the sample encoding features through the first initial encoding network, the sample encoding features, sample rhythm features and sample speech style features can also be processed through the first initial decoding network to obtain a sample predicted spectrum; obtain the sample spectrum of the sample audio data, and update the parameters in the first initial encoding network and the first initial decoding network based on the sample spectrum and the sample predicted spectrum; if the first initial encoding network after the parameter update and the first initial decoding network after the parameter update meet the first preset condition, the first initial encoding network after the parameter update is determined as the first encoding network, and the first initial decoding network after the parameter update is determined as the first decoding network. The sample predicted spectrum can be the spectrum corresponding to the sample predicted audio data. In other words, the neural network synthesizer can be called to synthesize the sample predicted spectrum into the sample predicted audio. The first preset condition is used to indicate that the parameters and structure of the first initial encoding network and the first initial decoding network have reached the target satisfaction level. The target satisfaction level can be used to indicate that the matching degree between the generated sample prediction spectrum and the sample audio spectrum meets the conditions. For example, the loss value between the sample prediction spectrum and the sample spectrum can be determined by a loss function. If the loss value satisfies the preset loss value condition, the first initial encoding network and the first initial decoding network after parameter update meet the first preset condition. The preset loss value condition can be that the loss value is less than the preset value, or the change between the loss values obtained after each update of the parameters of the first initial encoding network and the first initial decoding network is less than the preset value, which is not limited here. Among them, the preset loss value condition or preset value can be set by the model training user. The setting of the preset condition or preset value may be different for different models and scenarios, which is not limited here.
[0060] In a possible implementation, determining the source coding features of the source audio data through the first coding network may include the following steps: extracting at least one frame of audio included in the source audio data and the phonemes corresponding to each frame of audio through the first coding network, and constructing a source speech posterior probability matrix based on at least one frame of audio and the phonemes corresponding to each frame of audio; and performing dimension compression on the source speech posterior probability matrix through the first coding network to obtain the source coding features of the source audio data. Wherein, the horizontal axis of the speech posterior probability matrix represents the number of frames in the time dimension, and the vertical axis represents the category of each symbol, which can be a phoneme category. The speech posterior probability matrix can also be called a PPG matrix (Phonetic Posteriorgrams). For example, a piece of source audio data corresponding to the content is "v3 in1 h e2 ch eng2", and at least one burst of audio corresponding to the source audio data is extracted, then the factors corresponding to each frame of audio can be obtained as: "v3", "in1", "h", "e2", "ch", "eng2", then each frame of audio can be represented as a column vector, and the multiple column vectors are sorted in time order, then the PPG matrix can be obtained. The dimensional compression of the source speech posterior probability matrix through the first encoding network can be performed through the bottleneck layer of the first encoding network, so that the first encoding network can discard the source speech style features in the speech posterior probability matrix when encoding the source audio data, thereby achieving dimensional compression of the PPG matrix.
[0061] S103: extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and target speech style features to obtain second predicted audio data.
[0062] The source text data in the source audio data is used to indicate the linguistic features of the corresponding text content in the source audio data, and the linguistic features include but are not limited to Chinese phonemes, English phonemes, Chinese vowels, tones, word boundaries, phrase boundaries or sentence boundaries and other feature information. The source text data can usually be represented in the form of a phoneme sequence. Phonemes are the smallest speech units divided from the perspective of sound quality. Chinese phoneme sequences can usually be composed of vowels, vowels, and silent syllables, where vowels are tones. For example, if the corresponding text content in the source audio data is "speech synthesis", the tonal speech sequence corresponding to the text content is "v3 in1 h e2 ch eng2", so that the source text data can be obtained as "v3 in1 h e2 ch eng2". The initial speech synthesis model can be a variant of the tacotron2 system or other neural network models, which is not limited here.
[0063] In a possible implementation, the initial speech synthesis model includes a second initial encoding network and a second initial decoding network, and calling the initial speech synthesis model to process the source text data and the target voice style features may include the following steps: determining the text encoding features of the source text data through the second initial encoding network; processing the text encoding features and the target voice style features through the second initial decoding network to obtain the second predicted sound spectrum of the source text data under the voice style of the target audio data; and constructing the second predicted audio data based on the second predicted sound spectrum. Among them, the second initial encoding network is used to determine the text encoding features of the source audio data, and the text encoding features are the feature vectors or feature matrices of the source text data extracted by the second initial encoding network. If the second initial encoder converts the input phoneme sequence (i.e., the source text data) into context-related abstract text features, the second initial encoder is usually composed of a model with context-related relevance, such as a bidirectional recurrent neural network and a convolutional neural network, and the text encoding features obtained by the second initial encoding network have context-related characteristics. The second initial decoding network is used to receive the text encoding features and the target voice style features output by the second initial encoding network, thereby obtaining the second predicted sound spectrum. The second predicted sound spectrum may refer to a sound spectrum corresponding to the second predicted audio data, such as a mel spectrum, a spectrogram (spectrogram), a mel cepstrum, etc. When the second predicted audio data is constructed based on the second predicted sound spectrum, the second predicted sound spectrum may be input into a neural network synthesizer to obtain the second predicted audio data.
[0064] For example, see Figure 5 , Figure 5 It is a flow chart of a method for acquiring audio data provided in an embodiment of the present application. In this chart, after the source text data is input into the second initial encoding network, the text encoding features corresponding to the source text data can be obtained. The second initial encoding network processes the text encoding features and the target speech style features to obtain a second predicted spectrogram, and then determines the second predicted audio data based on the second predicted spectrogram.
[0065] In a possible implementation, the initial speech synthesis model may include a second initial encoding network, an initial attention mechanism, and a second initial decoding network. Then, calling the initial speech synthesis model to process the source text data and the target voice style features may include the following steps: determining the text encoding features of the source text data through the second initial encoding network; inputting the text encoding features and the target voice style features into the initial attention mechanism; processing the output result of the initial attention mechanism through the second initial decoding network to obtain the second predicted sound spectrum of the source text data under the voice style of the target audio data; and constructing the second predicted audio data based on the second predicted sound spectrum. Among them, the second initial encoding network is used to determine the text encoding features of the source audio data. Since the second predicted audio data is usually a monotonic mapping process from a phoneme sequence (i.e., the source text data) to a spectrum sequence when synthesized, the spectrum sequence can constitute the second predicted sound spectrum. Therefore, when generating each frame spectrum of the second predicted sound spectrum, it is only necessary to obtain the phoneme content corresponding to the frame spectrum. The initial attention mechanism is used to generate the phoneme content corresponding to each frame of the spectrum to be generated, so that the second initial decoder can better predict the next frame spectrum, thereby obtaining the second predicted sound spectrum. The second initial decoder network is used to receive the output result of the initial attention mechanism, and process the output result to obtain the second predicted sound spectrum. For example, the first decoding network receives the output of the initial attention mechanism at time t0 and the predicted spectrum of the adjacent frame before time t0 (i.e., the predicted spectrum of the previous frame) to predict the spectrum at time t0. In other words, the generation of the predicted spectrum of each frame needs to rely on the predicted spectrum of the previous frame, and then the predicted spectrum of each frame can be arranged in chronological order to obtain the second predicted sound spectrum. The second predicted audio data based on the second predicted sound spectrum can be constructed by a neural network synthesizer. For example, the neural network synthesizer processes each frame of frequency domain data of the second predicted sound spectrum to obtain multiple frames of time domain data corresponding to each frame of frequency domain data, thereby synthesizing the second predicted audio data according to the multiple frames of time domain data. For example, please refer to Figure 6 , Figure 6 This is a flow chart of a method for acquiring audio data in an embodiment of the present application. In this figure, after the source text data is input into the second initial encoding network, the text encoding features corresponding to the source text data can be obtained, and then the target speech style features and the text encoding features are input into the initial attention mechanism to obtain the output result of the initial attention mechanism. The second initial encoding network processes the output result to obtain a second predicted sound spectrum, and then determines the second predicted audio data based on the second predicted sound spectrum.
[0066] S104. Update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0067] Among them, the initial speech synthesis model can be updated by updating the parameters in the initial speech synthesis model according to the loss value between the first predicted audio data and the second predicted audio data, or by updating the parameters in the initial speech synthesis model according to the loss value between the first predicted sound spectrum corresponding to the first predicted audio data and the second predicted sound spectrum corresponding to the second predicted audio data. There is no limitation here.
[0068] In a possible implementation, updating the parameters in the initial speech synthesis model based on the loss value between the first predicted audio data and the second predicted audio data may include the following steps: calling a target loss function to determine the target loss value between the first predicted audio data and the second predicted audio data; if the target loss value satisfies a preset loss value condition, updating the parameters in the initial speech synthesis model based on the target loss value. Wherein, the loss function refers to the target to be minimized during the model training process, such as the mean square error loss function (MES loss), the mean absolute error loss function (MAE loss) or the cross entropy loss function, etc., which are not limited here. The preset loss value condition may be that the target loss value is greater than a preset value, or that the change between the target loss values obtained after each update of the parameters of the initial speech synthesis model is greater than a preset value, which is not limited here. Optionally, the first predicted audio data or the second predicted audio data may be represented by an audio signal, and calculating the target loss value between the first predicted audio data and the second predicted audio data may also be said to be calculating the loss value between the audio signals corresponding to the first predicted audio data and the second predicted audio data.
[0069] In a possible implementation, updating the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data may include the following steps: obtaining a first predicted sound spectrum corresponding to the first predicted audio data and a second predicted sound spectrum corresponding to the second predicted audio data; calling a target loss function to determine a target loss value between the first predicted sound spectrum and the second predicted sound spectrum; if the target loss value satisfies a preset loss value condition, updating the parameters in the initial speech synthesis model based on the target loss value. Wherein, the first predicted sound spectrum is determined according to the first predicted audio data, and the second predicted sound spectrum is determined according to the second predicted audio data, and the time domain information of the predicted audio data (the first predicted audio data or the second predicted audio data) can be converted into frequency domain information through Fourier transform to obtain the corresponding predicted sound spectrum (the first predicted sound spectrum or the second predicted sound spectrum). For example, when the predicted sound spectrum (the first predicted sound spectrum or the second predicted sound spectrum) of the audio data (the first predicted audio data or the second predicted audio data) is a Mel spectrum, the time domain information of the audio data can be converted into frequency domain information through Fourier transform, and the frequency domain information is subjected to Mel scale transform to obtain a Mel spectrum, i.e., a Mel spectrum. The loss function refers to the target to be minimized during the model training process, such as the mean square error loss function (MES loss), the mean absolute error loss function (MAE loss) or the cross entropy loss function, etc., which are not limited here. The preset loss value condition can be that the target loss value is greater than the preset value, or the change between the target loss values obtained after each update of the parameters of the initial speech synthesis model is greater than the preset value, which are not limited here.
[0070] The embodiment of the present application can obtain source audio data and target voice style features, call the voice conversion model to process the source audio data and the target voice style features, obtain first predicted audio data, thereby obtaining audio data corresponding to the target voice style features, call the initial voice synthesis model to process the source text data and the target voice style features, obtain second predicted audio data, and then update the parameters in the initial voice synthesis model based on the first predicted audio data and the second predicted audio data to train the initial voice synthesis model. This can help enhance the generalization ability of the trained voice synthesis model and improve the voice style similarity of the synthesized audio data.
[0071] See also Figure 7 , is a flow chart of a speech synthesis model training method provided by an embodiment of the present invention. The speech synthesis model training method can be executed by an electronic device. Figure 7 As shown, the speech synthesis model training method may include the following steps S701-S705.
[0072] S701: Obtain source audio data and target speech style features.
[0073] S702: Calling a speech conversion model to process source audio data and target speech style features to obtain first predicted audio data.
[0074] S703: extract source text data from the source audio data, and call the initial speech synthesis model to process the source text data and target speech style features to obtain second predicted audio data.
[0075] S704: Update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0076] Steps S701 - S704 may correspond to the relevant descriptions of steps S101 - S102 , which will not be repeated here.
[0077] It is understandable that the description of step S701-step S704 (step S101-step S102) is only one round of training in the speech synthesis model training process. When the updated initial speech synthesis model does not meet the second preset condition, the parameters in the initial speech synthesis model are updated again according to the first predicted audio data and the second predicted audio data. Optionally, the parameters of the updated initial speech synthesis model can be updated by another first predicted audio data obtained in step S702, and then by the other first predicted audio data, the corresponding source text data and the target speech style features, i.e., repeating steps S704-S704 until the updated initial speech synthesis model meets the second preset condition.
[0078] S705: If the initial speech synthesis model after parameter update meets the second preset condition, the initial speech synthesis model after parameter update is determined as the speech synthesis model.
[0079] Among them, the second preset condition is used to indicate that the parameters and structure of the initial speech synthesis model reach the target satisfaction level, and the target satisfaction level can be used to indicate that the matching degree between the second predicted audio data and the first predicted audio data meets the conditions, such as determining the target loss value between the second predicted audio data and the first predicted audio data through a loss function. If the target loss value satisfies the preset loss value condition, that is, the initial speech synthesis model satisfies the second preset condition. The preset loss value condition can be that the target loss value between the second predicted audio data generated by the initial speech synthesis model and the first predicted audio data is less than the preset value, or it can be that the change value between the target loss value between the second predicted audio data and the first predicted audio data generated by the initial speech synthesis model after parameter update is less than the preset value.
[0080] Optionally, the target satisfaction level can also be used to indicate that the degree of match between the second predicted sound spectrum and the first predicted sound spectrum meets a condition, such as determining a target loss value between the second predicted sound spectrum and the first predicted sound spectrum through a loss function. If the target loss value meets a preset loss value condition, that is, the initial speech synthesis model meets a second preset condition, the preset loss value condition can be that the target loss value between the second predicted sound spectrum generated by the initial speech synthesis model and the first predicted sound spectrum is less than a preset value, or that the change value between the target loss value between the second predicted sound spectrum generated by the initial speech synthesis model after parameter update and the first predicted sound spectrum is less than a preset value.
[0081] In a possible implementation, the initial speech synthesis model may include a second initial encoding network and a second initial decoding network, and the second initial encoding network may be determined as the second encoding network, and the second initial decoding network may be determined as the second decoding network.
[0082] In a possible implementation, the initial speech synthesis model may include a second initial encoding network, an initial attention mechanism, and a second initial decoding network. The second initial encoding network may be determined as the second encoding network, the initial attention mechanism may be determined as the attention mechanism, and the second initial decoding network may be determined as the second decoding network.
[0083] S706: Obtain text data to be synthesized.
[0084] The text data to be synthesized can usually be represented in the form of a phoneme sequence. For example, a text content such as "speech synthesis" can be obtained, and then the text content is processed to obtain the phoneme sequence "v3 in1 h e2 ch eng2" corresponding to the text content.
[0085] S707: If the initial speech synthesis model after parameter update meets the second preset condition, the initial speech synthesis model after parameter update is determined as the speech synthesis model.
[0086] The speech style feature of the target predicted audio data is the speech style indicated by the target speech style feature, and the audio content of the target predicted audio data is the content indicated by the text data to be synthesized. The speech synthesis model may include a second encoding network and a second decoding network, or the speech synthesis model may include a second encoding network, an attention mechanism, and a second decoding network.
[0087] In a possible implementation, the speech synthesis model may include a second encoding network and a second decoding network. Calling the speech synthesis model to process the text data to be synthesized and the target speech style features to obtain the target predicted audio data may include the following steps: determining the encoding features of the text data to be synthesized through the second encoding network; processing the encoding features of the text data to be synthesized and the target speech style features through the second decoding network to obtain the target predicted sound spectrum of the text data to be synthesized under the speech style of the target audio data; and constructing the target predicted audio data based on the target predicted sound spectrum. Among them, the second encoding network is used to determine the encoding features of the text to be synthesized, and the encoding features of the text to be synthesized are the feature vectors or feature matrices of the text data to be synthesized extracted by the second encoding network. The second decoding network is used to receive the encoding features of the text to be synthesized and the target speech style features output by the second encoding network, so as to obtain the target predicted sound spectrum. The target predicted sound spectrum may refer to the sound spectrum corresponding to the target predicted audio data, such as Mel spectrum, spectrogram (spectrogram), Mel cepstrum, etc. When constructing the target predicted audio data based on the target predicted sound spectrum, the target predicted sound spectrum can be input into the neural network synthesizer to obtain the target predicted audio data. For example, see Figure 8 , Figure 8 It is a flow chart of an audio data acquisition method provided by an embodiment of the present application. In this figure, after the text data to be synthesized is input into the second encoding network, the text encoding features to be synthesized corresponding to the text data to be synthesized can be obtained. The second encoding network processes the text encoding features to be synthesized and the target speech style features to obtain the target predicted sound spectrum, and then determines the target predicted audio data according to the target predicted sound spectrum. The speech style of the target predicted audio data is the speech style indicated by the target speech style features, and the audio content is the content indicated by the text data to be synthesized.
[0088] In a possible implementation, the initial speech synthesis model includes a second encoding network, an attention mechanism, and a second decoding network. Then, calling the speech synthesis model to process the text data to be synthesized and the target speech style features to obtain the target predicted audio data may include the following steps: determining the text encoding features to be synthesized of the text data to be synthesized through the second encoding network; inputting the text encoding features to be synthesized and the target speech style features into the attention mechanism; processing the output result of the attention mechanism through the second decoding network to obtain the target predicted sound spectrum of the text data to be synthesized under the speech style of the target audio data; and constructing the target predicted audio data based on the target predicted sound spectrum. Among them, the second encoding network is used to determine the text encoding features to be synthesized, and the text encoding features to be synthesized are the feature vectors or feature matrices of the text data to be synthesized extracted by the second encoding network. The attention mechanism is used to receive the text information to be synthesized and the target speech style features output by the second encoding network, and obtain an output result, which is used to indicate the phoneme content corresponding to the spectrum to be generated for each frame generated, so that the second initial decoder can better predict the spectrum of the next frame. The second decoding network is used to receive the output result of the attention mechanism, so as to obtain the target predicted sound spectrum. The target predicted sound spectrum may refer to the sound spectrum corresponding to the target predicted audio data, such as Mel spectrum, spectrogram (spectrogram), Mel cepstrum, etc. When the target predicted audio data is constructed based on the target predicted sound spectrum, the target predicted sound spectrum may be input into the neural network synthesizer to obtain the target predicted audio data. For example, see Fig. 9 , Fig. 9 This is a flow chart of a method for acquiring audio data in an embodiment of the present application. In this figure, after the text data to be synthesized is input into the second encoding network, the text encoding features to be synthesized corresponding to the text data to be synthesized can be obtained, and then the target speech style features and the text encoding features to be synthesized are input into the initial attention mechanism to obtain the output result of the initial attention mechanism. The second initial encoding network processes the output result to obtain the target predicted sound spectrum, and then determines the target predicted audio data according to the target predicted sound spectrum. The speech style of the target predicted audio data is the speech style indicated by the target speech style features, and the audio content is the content indicated by the text data to be synthesized.
[0089] The embodiment of the present application can obtain source audio data and target voice style features, call the voice conversion model to process the source audio data and the target voice style features, obtain the first predicted audio data, and then update the parameters in the initial voice synthesis model to train the initial voice synthesis model. When the updated initial voice synthesis model meets the second preset condition, the initial voice synthesis model with updated parameters is determined as the voice synthesis model to obtain the text data to be synthesized; the voice synthesis model is called to process the text data to be synthesized and the target voice style features to obtain the target predicted audio data. This can help enhance the generalization ability of the trained voice synthesis model and improve the voice style similarity of the synthesized target predicted audio data.
[0090] Based on the description of the above speech synthesis model training method embodiment, the present application embodiment also discloses a speech synthesis model training device, which can be configured in the above electronic device, for example, the device can be a computer program (including program code) running in the electronic device. The device can execute Figure 1 , 7 See the method shown in Fig.10 , the device can run the following modules:
[0091] An acquisition module 1001 is used to acquire source audio data and target voice style features, where the target voice style features are features corresponding to the voice style of the target audio data;
[0092] The processing module 1002 is used to call the speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0093] The processing module 1002 is further used to extract source text data from the source audio data, and call the initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0094] The processing module 1002 is further used to update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0095] In one implementation, the speech conversion model includes a first encoding network and a first decoding network, and the processing module 1002 is used to call the speech conversion model to process the source audio data and the target speech style feature to obtain the first predicted audio data, specifically for:
[0096] Acquire a source rhythm feature in the source audio data, and determine a source coding feature of the source audio data through a first encoding network, wherein the source rhythm feature is a feature corresponding to the rhythm in the source audio data;
[0097] Processing the source coding feature, the source rhythm feature and the target voice style feature through a first decoding network to obtain a first predicted spectrogram of the source audio data under the voice style of the target audio data;
[0098] First predicted audio data is constructed based on the first predicted sound spectrum.
[0099] In one implementation, before the processing module 1002 is used to determine the source coding feature of the source audio data through the first coding network, it is also used to:
[0100] Acquire sample audio data, sample rhythm features and sample voice style features, wherein the sample rhythm features are features corresponding to the rhythm in the sample audio data, and the sample voice style features are features corresponding to the voice style of the sample audio data;
[0101] Determining sample coding features of the sample audio data by a first initial coding network;
[0102] Processing the sample coding feature, the sample rhythm feature and the sample speech style feature through a first initial decoding network to obtain sample predicted audio data;
[0103] Update parameters in the first initial encoding network and the first initial decoding network based on the sample audio data and the sample predicted audio data;
[0104] If the first initial encoding network after parameter update and the first initial decoding network after parameter update meet the first preset condition, the first initial encoding network after parameter update is determined as the first encoding network, and the first initial decoding network after parameter update is determined as the first decoding network.
[0105] In one implementation, the initial speech synthesis model includes a second initial encoding network and a second initial decoding network, and the processing module 1002 is used to call the initial speech synthesis model to process the source text data and the target speech style feature to obtain second predicted audio data, specifically for:
[0106] Determining text encoding features of the source text data through a second initial encoding network;
[0107] Processing the text encoding features and the target voice style features through a second initial decoding network to obtain a second predicted spectrogram of the source text data under the voice style of the target audio data;
[0108] Second predicted audio data is constructed based on the second predicted sound spectrum.
[0109] In one implementation, when the processing module 1002 is used to update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data, it is specifically used to:
[0110] Obtaining a first predicted sound spectrum corresponding to the first predicted audio data, and a second predicted sound spectrum corresponding to the second predicted audio data;
[0111] Calling a target loss function to determine a target loss value between the first predicted sound spectrum and the second predicted sound spectrum;
[0112] If the target loss value meets a preset loss value condition, the parameters in the initial speech synthesis model are updated based on the target loss value.
[0113] In one implementation, when the processing module 1002 is used to determine the source coding feature of the source audio data through the first coding network, it is specifically used to:
[0114] Extracting at least one frame of audio included in the source audio data and the phonemes corresponding to each frame of audio through the first coding network, and constructing a source speech posterior probability matrix based on the at least one frame of audio and the phonemes corresponding to each frame of audio;
[0115] The first coding network is used to perform dimension compression on the source speech posterior probability matrix to obtain source coding features of the source audio data.
[0116] In one implementation, the processing module 1002 is used, after updating the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data, to:
[0117] If the initial speech synthesis model after parameter update meets the second preset condition, determining the initial speech synthesis model after parameter update as the speech synthesis model;
[0118] Obtain text data to be synthesized;
[0119] The speech synthesis model is called to process the text data to be synthesized and the target speech style features to obtain target predicted audio data.
[0120] Each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules, which is not limited in the present application.
[0121] The embodiment of the present application can obtain source audio data and target voice style features, call the voice conversion model to process the source audio data and the target voice style features, obtain first predicted audio data, thereby obtaining audio data corresponding to the target voice style features, call the initial voice synthesis model to process the source text data and the target voice style features, obtain second predicted audio data, and then update the parameters in the initial voice synthesis model based on the first predicted audio data and the second predicted audio data to train the initial voice synthesis model. This can help enhance the generalization ability of the trained voice synthesis model and improve the voice style similarity of the synthesized audio data.
[0122] See also Fig.11 , Fig.11 1 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. The electronic device in the embodiment of the present application includes a processor 1101 and a storage device 1102. Optionally, the electronic device may further include a network interface 1103. The processor 1101, the storage device 1102 and the network interface 1103 may exchange data. The network interface 1103 is controlled by the processor to send and receive messages. The storage device 1102 is used to store a computer program. The computer program includes program instructions. The processor 1101 is used to execute the program instructions stored in the storage device 1102. The processor 1101 is configured to call the program instructions to execute the above method.
[0123] The storage device 1102 may include a volatile memory, such as a random-access memory (RAM); the storage device 1102 may also include a non-volatile memory, such as a flash memory, a solid-state drive (SSD), etc.; the storage device 1102 may also include a combination of the above-mentioned types of memory.
[0124] The processor 1101 may be a central processing unit (CPU) 1101. In one embodiment, the processor 1101 may also be a graphics processing unit (GPU) 1101. The processor 1101 may also be a combination of a CPU and a GPU.
[0125] In one embodiment, the storage device 1102 is used to store program instructions. The processor 1101 can call the program instructions to perform the following steps:
[0126] In one implementation, the processor 1101 may also be configured to execute:
[0127] Acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data;
[0128] Calling a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data;
[0129] Extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data;
[0130] The parameters in the initial speech synthesis model are updated based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
[0131] In one implementation, the processor 1101 may also be configured to execute:
[0132] In one implementation, the speech conversion model includes a first encoding network and a first decoding network, and the processor 1101 is used to execute the calling of the speech conversion model to process the source audio data and the target speech style feature to obtain the first predicted audio data, specifically for executing:
[0133] Acquire a source rhythm feature in the source audio data, and determine a source coding feature of the source audio data through a first encoding network, wherein the source rhythm feature is a feature corresponding to the rhythm in the source audio data;
[0134] Processing the source coding feature, the source rhythm feature and the target voice style feature through a first decoding network to obtain a first predicted spectrogram of the source audio data under the voice style of the target audio data;
[0135] First predicted audio data is constructed based on the first predicted sound spectrum.
[0136] In one implementation, before the processor 1101 is used to determine the source coding feature of the source audio data by using the first coding network, it is also used to execute:
[0137] Acquire sample audio data, sample rhythm features and sample voice style features, wherein the sample rhythm features are features corresponding to the rhythm in the sample audio data, and the sample voice style features are features corresponding to the voice style of the sample audio data;
[0138] Determining sample coding features of the sample audio data by a first initial coding network;
[0139] Processing the sample coding feature, the sample rhythm feature and the sample speech style feature through a first initial decoding network to obtain sample predicted audio data;
[0140] Update parameters in the first initial encoding network and the first initial decoding network based on the sample audio data and the sample predicted audio data;
[0141] If the first initial encoding network after parameter update and the first initial decoding network after parameter update meet the first preset condition, the first initial encoding network after parameter update is determined as the first encoding network, and the first initial decoding network after parameter update is determined as the first decoding network.
[0142] In one implementation, the initial speech synthesis model includes a second initial encoding network and a second initial decoding network, and the processor 1101 is used to execute the calling of the initial speech synthesis model to process the source text data and the target speech style feature to obtain the second predicted audio data, specifically for executing:
[0143] Determining text encoding features of the source text data through a second initial encoding network;
[0144] Processing the text encoding features and the target voice style features through a second initial decoding network to obtain a second predicted spectrogram of the source text data under the voice style of the target audio data;
[0145] Second predicted audio data is constructed based on the second predicted sound spectrum.
[0146] In one implementation, when the processor 1101 is used to execute the updating of the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data, it is specifically used to execute:
[0147] Obtaining a first predicted sound spectrum corresponding to the first predicted audio data, and a second predicted sound spectrum corresponding to the second predicted audio data;
[0148] Calling a target loss function to determine a target loss value between the first predicted sound spectrum and the second predicted sound spectrum;
[0149] If the target loss value meets a preset loss value condition, the parameters in the initial speech synthesis model are updated based on the target loss value.
[0150] In one implementation, when the processor 1101 is used to perform the step of determining the source coding feature of the source audio data by using the first coding network, it is specifically used to perform:
[0151] Extracting at least one frame of audio included in the source audio data and the phonemes corresponding to each frame of audio through the first coding network, and constructing a source speech posterior probability matrix based on the at least one frame of audio and the phonemes corresponding to each frame of audio;
[0152] The first coding network is used to perform dimension compression on the source speech posterior probability matrix to obtain source coding features of the source audio data.
[0153] In one implementation, the processor 1101 is configured to, after updating the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data, further perform:
[0154] If the initial speech synthesis model after parameter update meets the second preset condition, determining the initial speech synthesis model after parameter update as the speech synthesis model;
[0155] Obtain text data to be synthesized;
[0156] The speech synthesis model is called to process the text data to be synthesized and the target speech style features to obtain target predicted audio data.
[0157] In a specific implementation, the device, processor 1101, storage device 1102, etc. described in the embodiments of the present application can execute the implementation method described in the above method embodiments, and can also execute the implementation method described in the embodiments of the present application, which will not be repeated here.
[0158] In an embodiment of the present application, a computer (readable) storage medium is also provided, wherein the computer storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, some or all of the steps executed in the above method embodiment can be executed. Optionally, the computer storage medium can be volatile or non-volatile.
[0159] The embodiment of the present application also provides a computer program product or computer program, which includes program instructions, which can be stored in a computer-readable storage medium. The processor of the computer device reads the program instructions from the computer-readable storage medium, and the processor executes the program instructions, so that the computer executes some or all of the steps executed in the above method, which will not be repeated here.
[0160] The term "multiple" as used herein refers to two or more than two. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.
[0161] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer storage medium, which can be a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.
[0162] The above disclosure is only part of the embodiments of the present application, which certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiments and equivalent changes made according to the claims of the present application are still within the scope covered by the present application.
Claims
1. A speech synthesis model training method, characterized in that: include: Acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data; Calling a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data; Extracting source text data from the source audio data, and calling an initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data; The parameters in the initial speech synthesis model are updated based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
2. The method according to claim 1, characterized in that The speech conversion model includes a first encoding network and a first decoding network, and the calling of the speech conversion model processes the source audio data and the target speech style feature to obtain first predicted audio data, including: Acquire a source rhythm feature in the source audio data, and determine a source coding feature of the source audio data through a first encoding network, wherein the source rhythm feature is a feature corresponding to the rhythm in the source audio data; Processing the source coding feature, the source rhythm feature and the target voice style feature through a first decoding network to obtain a first predicted spectrogram of the source audio data under the voice style of the target audio data; First predicted audio data is constructed based on the first predicted sound spectrum.
3. The method according to claim 2, characterized in that Before determining the source coding feature of the source audio data by the first coding network, the method further includes: Acquire sample audio data, sample rhythm features and sample voice style features, wherein the sample rhythm features are features corresponding to the rhythm in the sample audio data, and the sample voice style features are features corresponding to the voice style of the sample audio data; Determining sample coding features of the sample audio data by a first initial coding network; Processing the sample coding feature, the sample rhythm feature and the sample speech style feature through a first initial decoding network to obtain sample predicted audio data; Update parameters in the first initial encoding network and the first initial decoding network based on the sample audio data and the sample predicted audio data; If the first initial encoding network after parameter update and the first initial decoding network after parameter update meet the first preset condition, the first initial encoding network after parameter update is determined as the first encoding network, and the first initial decoding network after parameter update is determined as the first decoding network.
4. The method according to claim 1, characterized in that The initial speech synthesis model includes a second initial encoding network and a second initial decoding network, and the calling of the initial speech synthesis model to process the source text data and the target speech style feature to obtain second predicted audio data includes: Determining text encoding features of the source text data through a second initial encoding network; Processing the text encoding features and the target voice style features through a second initial decoding network to obtain a second predicted spectrogram of the source text data under the voice style of the target audio data; Second predicted audio data is constructed based on the second predicted sound spectrum.
5. The method according to claim 1, characterized in that: The updating of the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data comprises: Obtaining a first predicted sound spectrum corresponding to the first predicted audio data, and a second predicted sound spectrum corresponding to the second predicted audio data; Calling a target loss function to determine a target loss value between the first predicted sound spectrum and the second predicted sound spectrum; If the target loss value meets a preset loss value condition, the parameters in the initial speech synthesis model are updated based on the target loss value.
6. The method according to claim 2, characterized in that: The determining of the source coding feature of the source audio data by the first coding network comprises: Extracting at least one frame of audio included in the source audio data and the phonemes corresponding to each frame of audio through the first coding network, and constructing a source speech posterior probability matrix based on the at least one frame of audio and the phonemes corresponding to each frame of audio; The first coding network is used to perform dimension compression on the source speech posterior probability matrix to obtain source coding features of the source audio data.
7. The method according to claim 1, characterized in that: After updating the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data, the method further includes: If the initial speech synthesis model after parameter update meets the second preset condition, determining the initial speech synthesis model after parameter update as the speech synthesis model; Obtain text data to be synthesized; The speech synthesis model is called to process the text data to be synthesized and the target speech style features to obtain target predicted audio data.
8. A speech synthesis device, characterized in that: include: An acquisition module, used to acquire source audio data and target voice style features, wherein the target voice style features are features corresponding to the voice style of the target audio data; A processing module, configured to call a speech conversion model to process the source audio data and the target speech style feature to obtain first predicted audio data; The processing module is further used to extract source text data from the source audio data, and call the initial speech synthesis model to process the source text data and the target speech style features to obtain second predicted audio data; The processing module is also used to update the parameters in the initial speech synthesis model based on the first predicted audio data and the second predicted audio data to train the initial speech synthesis model.
9. An electronic device, characterized in that: The invention comprises a processor and a storage device, wherein the processor and the storage device are connected to each other, wherein the storage device is used to store computer program instructions, and the processor is configured to execute the program instructions to implement the method according to any one of claims 1 to 7.
10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, they are used to execute the method according to any one of claims 1 to 7.
