Speech synthesis model training method and device, speech synthesis method and device and storage medium

Abstract

The invention provides a speech synthesis model training method, a speech synthesis method, a speech synthesis device and a storage medium, and the speech synthesis model training method comprises the steps: obtaining the sample text data of a plurality of speakers and the acoustic features of the sample audio data of the plurality of speakers according to the sample audio data of the plurality of speakers, feature extraction is carried out on the sample text data of each speaker to obtain sample text features of each speaker, a speech synthesis sample of each speaker is constructed according to the sample text features of each speaker and acoustic features of the sample audio data of each speaker, and speech synthesis is carried out according to the speech synthesis samples of multiple speakers. And respectively carrying out model training to obtain speech synthesis models of the plurality of speakers. Automatic training of the speech synthesis model is carried out in a one-button mode, and the iteration efficiency of the speech synthesis technology is improved.

Description

Technical Field

[0001] The present application relates to the field of speech processing technology, and in particular to a speech synthesis model training method, a speech synthesis method, a device and a storage medium. Background Art

[0002] With the development of deep learning technology, speech synthesis has not only achieved great improvements in generation performance, but also made rapid progress in application fields. Speech synthesis technology is increasingly used in various fields. Among them, speech synthesis is a technology that uses text input and other conditions such as emotion, emotion intensity, style and other information as conditions to synthesize natural and expressive speech.

[0003] Currently, in the process of speech synthesis, maintenance personnel are often required to perform compliance checks, data processing and feature extraction on the data, send it to the speech synthesis model for training, and then save and export the trained model.

[0004] However, manual operations are prone to errors, consume a lot of time and energy, and are not conducive to product iteration or algorithm iteration. Summary of the invention

[0005] In view of this, the embodiments of the present application provide a speech synthesis model training method, a speech synthesis method, a device and a storage medium to solve the problem that manual operation is prone to errors, consumes a lot of time and energy, and is not conducive to product iteration or algorithm iteration.

[0006] In a first aspect, an embodiment of the present application provides a method for training a speech synthesis model, comprising:

[0007] According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers;

[0008] Performing feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker;

[0009] Constructing a speech synthesis sample of each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0010] Model training is performed respectively according to the speech synthesis samples of the multiple speakers to obtain speech synthesis models of the multiple speakers.

[0011] In a second aspect, the present application also provides a speech synthesis method, including:

[0012] Obtain the text to be processed and the identification of the target speaker;

[0013] Determining a target speech synthesis model from speech synthesis models of multiple speakers according to the identifier of the target speaker, wherein the speech synthesis models of the multiple speakers are trained using the speech synthesis model training method according to any one of the first aspects;

[0014] Performing feature extraction on the text to be processed to obtain text features of the text to be processed;

[0015] Processing the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features;

[0016] According to the acoustic features corresponding to the text features, synthesized speech is obtained.

[0017] In a third aspect, the present application also provides a speech synthesis model training device, including:

[0018] An acquisition module, used to acquire sample text data of the multiple speakers and acoustic features of the sample audio data of the multiple speakers respectively according to the sample audio data of the multiple speakers;

[0019] A feature extraction module is used to extract features from the sample text data of each speaker to obtain the sample text features of each speaker;

[0020] A construction module, used to construct a speech synthesis sample of each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0021] The training module is used to perform model training respectively according to the speech synthesis samples of the multiple speakers to obtain the speech synthesis models of the multiple speakers.

[0022] In a fourth aspect, the present application also provides a speech synthesis device, including:

[0023] An acquisition module, used to acquire the text to be processed and the timbre identifier of the target speaker;

[0024] A determination module, configured to determine a target speech synthesis model from speech synthesis models of multiple speakers according to the timbre identifier of the target speaker, wherein the speech synthesis models of the multiple speakers are obtained by training using the speech synthesis model described in any one of the first aspect;

[0025] A feature extraction module, used to extract features of the text to be processed to obtain text features of the text to be processed;

[0026] A processing module, used for processing the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features;

[0027] The acquisition module is also used to obtain synthesized speech according to the acoustic features corresponding to the text features.

[0028] In the fifth aspect, an embodiment of the present application also provides an electronic device, comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the memory through the bus, and the processor executes the machine-readable instructions to execute the speech synthesis model training method described in any one of the first aspect or the speech synthesis method described in the second aspect.

[0029] In a sixth aspect, an embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the speech synthesis model training method described in any one of the first aspect or the speech synthesis method described in the second aspect is executed.

[0030] The present application provides a speech synthesis model training method, a speech synthesis method, a device and a storage medium, wherein the speech synthesis model training method comprises: obtaining sample text data of multiple speakers and acoustic features of sample audio data of multiple speakers respectively according to sample audio data of multiple speakers, extracting features of sample text data of each speaker to obtain sample text features of each speaker, constructing speech synthesis samples of each speaker according to sample text features of each speaker and acoustic features of sample audio data of each speaker, and performing model training respectively according to speech synthesis samples of multiple speakers to obtain speech synthesis models of multiple speakers. The iteration efficiency of speech synthesis technology is improved by performing automatic training of speech synthesis models in a one-click manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present application and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0032] Figure 1 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 1 ;

[0033] Figure 2Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 2 ;

[0034] Figure 3 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 3 ;

[0035] Figure 4 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 4 ;

[0036] Figure 5 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 5 ;

[0037] Figure 6 A flowchart of a speech synthesis method provided in an embodiment of the present application;

[0038] Figure 7 A schematic diagram of the structure of a speech synthesis model training device provided in an embodiment of the present application;

[0039] Figure 8 A schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application;

[0040] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0041] To make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application claimed for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present application.

[0042] Speech synthesis technology is increasingly being used in various fields, such as virtual humans, AI dubbing, intelligent non-player characters (NPCs) in games, and voice navigation. With the popularity of these applications, an important issue has emerged in the eyes of developers and researchers, that is, how to quickly and conveniently apply new speaker data to speech synthesis models. The traditional method is usually for speech synthesis technology maintenance personnel to perform compliance testing, data processing, and feature extraction on the data. After the data processing is completed, the data is added to the speech synthesis database, the database information is regenerated, and finally the database information is sent to the speech synthesis model for training, and the trained model is saved and exported. It can be seen that the process of the traditional method is very lengthy and cumbersome, and manual operation is prone to errors, which consumes a lot of time and energy and is not conducive to product iteration or algorithm iteration.

[0043] Based on this, the present application improves the iteration efficiency of speech synthesis technology by performing automated training of the speech synthesis model in a one-click manner.

[0044] Figure 1 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 1 The execution subject of this embodiment may be an electronic device, such as a terminal device, a cloud server, etc.

[0045] like Figure 1 As shown, the method may include:

[0046] S101. Acquire sample text data of the multiple speakers and acoustic features of the sample audio data of the multiple speakers respectively according to sample audio data of the multiple speakers.

[0047] The sample audio data is the speech data of the speaker. The sample audio data of multiple speakers are obtained, and based on the sample audio data of each speaker, the sample text data and the acoustic features of the sample audio data of each speaker are obtained. The number of sample audio data of each speaker is multiple, and one sample audio data corresponds to one sample text data.

[0048] Among them, the sample text data can be text data determined based on the sample audio data, and the sample text data includes: text information of the sample audio data, such as audio text information and voice text information of the sample audio data, the audio text information is text information related to the audio text of the sample audio data, and the voice text information is text information related to the audio pronunciation of the sample audio data.

[0049] The acoustic features of the sample audio data of each speaker may include at least one of a Mel spectrum, a linear spectrum, and a fundamental frequency of the sample audio data.

[0050] S102: Extract features from the sample text data of each speaker to obtain sample text features of each speaker.

[0051] The number of sample text features for each speaker includes but is not limited to one. Feature extraction is performed on the sample text data of each speaker to obtain the sample text features of each speaker. The sample text features are used to indicate the text features of the sample audio data, for example, they may include audio text features and voice text features of the sample audio data.

[0052] Among them, the audio text feature can be determined based on the audio text information, and the voice text feature can be determined based on the voice text information. For example, the audio text feature is the audio text information, and the voice text feature is the voice text information.

[0053] S103: construct a speech synthesis sample for each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker.

[0054] S104: Perform model training respectively according to the speech synthesis samples of the multiple speakers to obtain speech synthesis models of the multiple speakers.

[0055] According to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker, a speech synthesis sample of each speaker is constructed. The speech synthesis sample includes: the sample text features of each speaker and the acoustic features of the sample audio data of each speaker.

[0056] Then, based on the speech synthesis samples of multiple speakers, model training is performed respectively to obtain speech synthesis models for multiple speakers. During the model training process, the acoustic features of the sample audio data of each speaker are used as model annotation information, and the sample text features of each speaker are used as model input. After being processed by a preset model, the model output acoustic features of the sample audio data of each speaker are obtained. Then, based on the acoustic features of the sample audio data of each speaker and the model output acoustic features, a technical loss function is used, and model training is performed according to the loss function to obtain a speech synthesis model for each speaker.

[0057] In some embodiments, during model training, the model obtained when the preset iteration conditions are met can be used as the speaker's speech synthesis model, and the preset iteration stop conditions include reaching a preset number of iterations and the loss function being less than a preset loss value.

[0058] It is worth noting that the terminal device performs feature extraction on the sample text data of each speaker. After obtaining the sample text features of each speaker, it can also store the sample text features of each speaker and the acoustic features of the sample audio data of each speaker in a test file corresponding to the file name according to the file name of the sample audio data, and add the file corresponding to the file name to the sound library. Then, the test file of each speaker in the sound library is uploaded to the cloud service platform. On the cloud service platform, the test file is read using the speech synthesis training framework to perform model training. After each round of training is completed, the model can be automatically saved. After reaching the preset number of iterations, the terminal device can also download the latest saved model from the cloud service platform and save it to the corresponding location.

[0059] In the speech synthesis model training method of the present embodiment, based on the sample audio data of multiple speakers, sample text data of multiple speakers and acoustic features of sample audio data of multiple speakers are obtained respectively, feature extraction is performed on the sample text data of each speaker to obtain the sample text features of each speaker, speech synthesis samples of each speaker are constructed based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker, and model training is performed based on the speech synthesis samples of multiple speakers to obtain speech synthesis models of multiple speakers. The iteration efficiency of speech synthesis technology is improved by performing automatic training of speech synthesis models in a one-click manner.

[0060] Figure 2 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 2 ,like Figure 2 As shown, in an optional implementation, step S101, based on sample audio data of multiple speakers, respectively obtaining sample text data of multiple speakers and acoustic features of sample audio data of multiple speakers, may include:

[0061] S201. Generate at least two types of text information corresponding to the sample audio data of each speaker according to the sample audio data of each speaker.

[0062] The at least two types of text information include: audio text information and voice text information. The audio text information is text information related to the audio text of the sample audio data, and the voice text information is text information related to the audio pronunciation of the sample audio data.

[0063] The utterance text information includes at least one of the pronunciation annotation text information, rhythm annotation text information, pronunciation element text information and pronunciation element duration information of the sample audio data. The pronunciation annotation text information is used to represent the phonetic notation of the sample audio data, the pronunciation element text information is used to represent the phonemes decomposed from the phonetic notation, the pronunciation element duration information is used to represent the duration of the phonemes, and the rhythm annotation text information is used to represent the rhythmic regularity of the text of the sample audio data.

[0064] S202: Generate sample text data for each speaker based on at least two types of text information.

[0065] The sample text data of each speaker includes at least two types of text information corresponding to the sample audio data of each speaker.

[0066] In some embodiments, the sample text data may be generated according to a preset data specification, which may be, for example, 7 lines of [file name.wav] [audio corresponding text] [phonetic notation] [rhythm annotation] [phoneme] [phoneme duration] [blank line], wherein [file name.wav] indicates that the file name of the sample audio data and the format is wav, [audio corresponding text] indicates the audio text information of the sample audio data, and [phonetic notation] [rhythm annotation] [phoneme] [phoneme duration] respectively indicate the pronunciation annotation text information, rhythm annotation text information, pronunciation element text information and pronunciation element duration information of the sample audio data.

[0067] In some embodiments, phonetic information, prosodic annotation text information, pronunciation element text information and pronunciation element duration information can be separated by spaces, and unannotated lines include blank lines. In addition, compliance checks can be performed on sample text data and sample audio data. Compliance checks on sample text data include but are not limited to: checking whether the number of lines of sample text data is correct to determine whether there are missing lines, and then determining whether information is missing; checking whether the number of phonetic symbols, prosodic annotations, phonemes, and phoneme durations separated by spaces is correct to determine whether information is missing; checking whether the number of sample text data and sample audio data is consistent to determine whether sample text data is missing; compliance checks on sample audio data include but are not limited to checking whether the audio sampling rate of the sample audio data is correct, for example, whether it is 48khz.

[0068] S203: extract features from the sample audio data of each speaker to obtain acoustic features of the sample audio data of each speaker.

[0069] Acoustic features are extracted from the sample audio data of each speaker to obtain acoustic features of the sample audio data of each speaker. The acoustic features may include at least one of a Mel spectrum, a linear spectrum, and a fundamental frequency of the sample audio data.

[0070] In the speech synthesis model training method of this embodiment, the acoustic features of the sample text data of each speaker and the sample audio data of each speaker are automatically extracted with one click, which improves the iteration efficiency of the speech synthesis technology and reduces labor costs and time costs.

[0071] Figure 3 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 3 ,like Figure 3 As shown, in an optional implementation, step S201, based on the sample audio data of each speaker, generates at least two types of text information corresponding to the sample audio data of each speaker, which may include:

[0072] S301, converting sample audio data of each speaker into text to generate audio text information.

[0073] The sample audio data of each speaker is converted into text to generate audio text information converted from the sample audio data, where the audio text information is text information related to the audio text of the sample audio data.

[0074] S302: Generate spoken text information based on the audio text information and the sample audio data of each speaker.

[0075] The audio text information and the sample audio data of each speaker are combined to generate the voiced text information, which is the text information related to the audio pronunciation of the sample audio data. That is, the voiced text information is determined comprehensively based on the audio text information and the sample audio data.

[0076] In the speech synthesis model training method of this embodiment, audio text information is generated by converting the sample audio data of each speaker into text, and voiced text information is generated based on the audio text information and the sample audio data of each speaker. The one-click automatic extraction of audio text information and voiced text information improves the iteration efficiency of speech synthesis technology and reduces labor costs and time costs.

[0077] Figure 4 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 4 ,like Figure 4 As shown, in an optional implementation, step S302, generating voiced text information according to the audio text information and the sample audio data of each speaker, may include:

[0078] S401. Generate pronunciation annotation text information and pronunciation element text information according to audio text information.

[0079] The audio text information is the audio text of the sample audio data. The audio text information is pronounced and annotated to generate pronunciation annotation text information and pronunciation element text information. The pronunciation element is obtained by decomposing the pronunciation annotation. For example, if the audio text information is "Today is Friday", the pronunciation annotation text information is "jin tian xing qi wu" and the pronunciation element text information is "jintianxingqiwu".

[0080] S402: Extract pronunciation rhythm from sample audio data of each speaker to obtain rhythm-annotated text information.

[0081] The pronunciation rhythm is extracted based on the audio text information and the sample audio data of each speaker to obtain the rhythm-tagged text information. The rhythm-tagged text information is used to characterize the rhythmic regularity of the audio text of the sample audio data. That is to say, the pronunciation rhythm of the audio text corresponding to the audio text information is extracted in combination with the sample audio data to obtain the rhythm-tagged text information of the audio text.

[0082] S403: Generate pronunciation element duration information according to the audio text information and the sample audio data of each speaker.

[0083] The pronunciation element duration information is used to indicate the duration of the phoneme. The duration of the sample audio data is obtained. Based on the duration of the sample audio data, the duration of the audio text corresponding to the audio text information can be determined. Then, based on the duration of the audio text, the duration of the phoneme of the audio text, that is, the pronunciation element duration, can be determined.

[0084] The spoken text information includes: pronunciation annotation text information, pronunciation element text information, rhythm annotation text information and pronunciation element duration information.

[0085] In an optional implementation, step S403, generating pronunciation element duration information according to the audio text information and the sample audio data of each speaker, may include:

[0086] The initial pronunciation element duration information is generated based on the audio text information and the sample audio data of each speaker; if the initial pronunciation element duration information is a preset null value, the preset alignment tool is used to align the audio text information and the sample audio data of each speaker, and generate the pronunciation element duration information.

[0087] Based on the audio text information and the sample audio data of each speaker, the initial pronunciation element duration information is generated. If the initial pronunciation element duration information is a preset null value, that is, the initial pronunciation element duration information is not determined, the preset alignment tool can be used to align the audio text information with the sample audio data of each speaker to obtain an alignment result. According to the alignment result, the duration of the phonemes of the audio text can be determined.

[0088] The sample audio data and the audio text information are aligned, and the time end of the pronunciation element for the sample audio data can be obtained from the alignment result, thereby obtaining the duration information of the pronunciation element, and the alignment is preset. The preset alignment tool can be, for example, a Montreal Forced Aligner (MFA) tool.

[0089] In some embodiments, initial pronunciation element duration information may be generated based on the input alignment operation, the audio text information, and the sample audio data of each speaker.

[0090] In the speech synthesis model training method of the present embodiment, pronunciation annotation text information, pronunciation element text information, rhythm annotation text information and pronunciation element duration information are automatically extracted with one click, thereby improving the iteration efficiency of the speech synthesis technology.

[0091] Figure 5 Schematic diagram of the process of the speech synthesis model training method provided in the embodiment of the present application Figure 5 ,like Figure 5 As shown, in an optional implementation, the speech synthesis sample includes: a speech synthesis training sample and a speech synthesis test sample; step S104, according to the speech synthesis samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers, including:

[0092] S501 . Perform model training respectively according to speech synthesis training samples of multiple speakers to obtain speech synthesis models of the multiple speakers.

[0093] The method may further include:

[0094] S502: According to the speech synthesis test samples of the multiple speakers, perform model tests on the speech synthesis models of the multiple speakers respectively.

[0095] The speech synthesis model of each speaker is tested according to the speech synthesis test sample of each speaker. If the test result meets the preset conditions, the training of the speech synthesis model is stopped. If the test result does not meet the preset conditions, the model training is continued according to the speech synthesis training sample of each speaker. The process is repeated until the test result meets the preset conditions.

[0096] It is worth noting that after the terminal device extracts features from the sample text data of each speaker and obtains the sample text features of each speaker, it can also determine the training sample text features and training acoustic features, as well as the test sample text features and test acoustic features from the sample text features of each speaker and the acoustic features of the sample audio data of each speaker according to the file name of the sample audio data, and then store the training sample text features and training acoustic features in the training file corresponding to the file name, and store the training sample text features and training acoustic features in the test file corresponding to the file name, and store the test file and the training file in the timbre library, so that the timbre library includes test files and training files corresponding to multiple speakers, and then generate a training file ID list, a test file ID list and a timbre ID list in the timbre library, and establish a corresponding relationship between the timbre ID list, the training file ID list and the test file ID list, wherein the timbre ID list includes the timbre identifiers of multiple speakers, and different speakers correspond to different timbre identifiers.

[0097] Afterwards, according to the timbre identification of each speaker, the corresponding training file is selected from the training file ID list of the timbre library and uploaded to the cloud service platform. On the cloud service platform, the test file is read using the speech synthesis training framework to perform model training. After each round of training is completed, the model can be automatically saved. After reaching the preset number of iterations, the terminal device can also download the latest saved speech synthesis model of each speaker from the cloud service platform and save it to the corresponding location. Then, according to the timbre identification of each speaker, the corresponding test file is determined from the test file ID list of the timbre library, and the model is tested according to the corresponding test file until the test result meets the preset conditions.

[0098] In summary, the speech synthesis model training process can be divided into six steps: data compliance check, feature extraction, data upload, model training, model preservation, and effect verification. These six steps can be integrated into a speech synthesis automatic training tool, so as to achieve one-click start and autonomous end of model training, and standardize each step, which improves the iteration efficiency of speech synthesis technology and reduces labor and time costs.

[0099] Figure 6 This is a flow chart of a speech synthesis method provided in an embodiment of the present application. The executor of this embodiment may be an electronic device, such as a terminal device, a cloud server, etc.

[0100] like Figure 6 As shown, the method may include:

[0101] S601: Obtain the text to be processed and the identifier of the target speaker.

[0102] S602: Determine a target speech synthesis model from speech synthesis models of multiple speakers according to the identifier of the target speaker.

[0103] The identifier of the target speaker may be an identity identifier or a timbre identifier of the target speaker.

[0104] According to the identifier of the target speaker, a target speech synthesis model of the target speaker is queried from the speech synthesis models of multiple speakers to determine the target speech synthesis model, and the speech synthesis models of the multiple speakers are trained using the above-mentioned speech synthesis model training method.

[0105] S603: Extract features of the text to be processed to obtain text features of the text to be processed.

[0106] The text features of the text to be processed may include audio-text features and spoken text features of the text to be processed. The audio-text features may be audio-text information of the text to be processed, and the audio-text information may be text information related to the audio-text of the text to be processed. For example, if the text to be processed is “Today is Monday”, then the audio-text features may be “Today is Monday”.

[0107] The spoken text feature may include the spoken text information of the text to be processed, and the spoken text information includes at least one of pronunciation annotation text information, rhythm annotation text information, pronunciation element text information and pronunciation element duration information.

[0108] S604: Process the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features.

[0109] S605: Obtain synthesized speech according to the acoustic features corresponding to the text features.

[0110] The text features of the text to be processed are used as the input of the target speech synthesis model, and the target speech synthesis model is used to process the text features of the text to be processed to obtain acoustic features corresponding to the text features, and then speech synthesis is performed according to the acoustic features corresponding to the text features to obtain synthesized speech, wherein the style of the synthesized speech is close to the speech style of the target speaker corresponding to the target speech synthesis model.

[0111] In the speech synthesis method of this embodiment, according to the identification of the target speaker, a target speech synthesis model is determined from the speech synthesis models of multiple speakers, feature extraction is performed on the text to be processed to obtain text features of the text to be processed, the text features of the text to be processed are processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features, and synthesized speech is obtained according to the acoustic features corresponding to the text features. The iteration efficiency of the speech synthesis technology is improved.

[0112] Figure 7This is a structural schematic diagram of a speech synthesis model training device provided in an embodiment of the present application, which can be integrated into an electronic device.

[0113] like Figure 7 As shown, the device may include:

[0114] An acquisition module 701 is used to acquire sample text data of multiple speakers and acoustic features of sample audio data of multiple speakers respectively according to sample audio data of multiple speakers;

[0115] A feature extraction module 702 is used to extract features from the sample text data of each speaker to obtain sample text features of each speaker;

[0116] A construction module 703, configured to construct a speech synthesis sample for each speaker based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0117] The training module 704 is used to perform model training according to the speech synthesis samples of multiple speakers, so as to obtain the speech synthesis models of the multiple speakers.

[0118] In an optional implementation, the acquisition module 701 is specifically configured to:

[0119] Generating at least two types of text information corresponding to the sample audio data of each speaker according to the sample audio data of each speaker; the at least two types of text information include: audio text information and voice text information;

[0120] Generate sample text data for each speaker based on at least two types of text information;

[0121] Feature extraction is performed on the sample audio data of each speaker to obtain the acoustic features of the sample audio data of each speaker.

[0122] In an optional implementation, the acquisition module 701 is specifically configured to:

[0123] Convert the sample audio data of each speaker into text to generate audio text information;

[0124] The spoken text information is generated based on the audio text information and the sample audio data of each speaker.

[0125] In an optional implementation, the acquisition module 701 is specifically configured to:

[0126] Generate pronunciation annotation text information and pronunciation element text information according to the audio text information;

[0127] Extracting pronunciation rhythm according to the audio text information and the sample audio data of each speaker to obtain rhythm annotation text information;

[0128] The pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker; wherein the voiced text information includes: pronunciation annotation text information, pronunciation element text information, rhythm annotation text information and pronunciation element duration information.

[0129] In an optional implementation, the acquisition module 701 is specifically configured to:

[0130] Generate initial pronunciation element duration information based on audio text information and sample audio data of each speaker;

[0131] If the initial pronunciation element duration information is a preset null value, a preset alignment tool is used to align the audio text information with the sample audio data of each speaker, and generate the pronunciation element duration information.

[0132] In an optional implementation, the speech synthesis sample includes: a speech synthesis training sample and a speech synthesis test sample; the training module 703 is specifically used to:

[0133] According to speech synthesis training samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers;

[0134] The device also includes:

[0135] The testing module 705 is used to perform model testing on the speech synthesis models of the multiple speakers respectively according to the speech synthesis test samples of the multiple speakers.

[0136] In the speech synthesis model training device of this embodiment, the acquisition module is used to respectively acquire the sample text data of multiple speakers and the acoustic features of the sample audio data of multiple speakers according to the sample audio data of multiple speakers, the feature extraction module is used to extract features from the sample text data of each speaker to obtain the sample text features of each speaker, the construction module is used to construct the speech synthesis sample of each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker, and the training module is used to respectively perform model training according to the speech synthesis samples of multiple speakers to obtain the speech synthesis models of multiple speakers. By performing automatic training of the speech synthesis model in a one-click manner, the iteration efficiency of the speech synthesis technology is improved.

[0137] Figure 8 This is a schematic diagram of the structure of a speech synthesis device provided in an embodiment of the present application, which can be integrated into an electronic device.

[0138] like Figure 8 As shown, the device may include:

[0139] An acquisition module 801 is used to acquire the text to be processed and the timbre identifier of the target speaker;

[0140] A determination module 802 is used to determine a target speech synthesis model from speech synthesis models of multiple speakers according to the timbre identifier of the target speaker, where the speech synthesis models of the multiple speakers are trained using the above-mentioned speech synthesis models;

[0141] A feature extraction module 803 is used to extract features of the text to be processed to obtain text features of the text to be processed;

[0142] The processing module 804 is used to process the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features;

[0143] The acquisition module 801 is further used to obtain synthesized speech according to the acoustic features corresponding to the text features.

[0144] In the speech synthesis device of this embodiment, the acquisition module is used to acquire the timbre identification of the text to be processed and the target speaker, the determination module is used to determine the target speech synthesis model from the speech synthesis models of multiple speakers according to the timbre identification of the target speaker, and the speech synthesis models of the multiple speakers are obtained by training with the above-mentioned speech synthesis model, the feature extraction module is used to extract features of the text to be processed to obtain text features of the text to be processed, the processing module is used to process the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features, and the acquisition module is used to obtain synthesized speech according to the acoustic features corresponding to the text features. The iteration efficiency of the speech synthesis technology is improved.

[0145] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, the device may include: a processor 901, a memory 902 and a bus 903, the memory 902 stores machine-readable instructions executable by the processor 901, when the electronic device is running, the processor 901 communicates with the memory 902 through the bus 903, and the processor 901 executes the machine-readable instructions to perform the following speech synthesis model training method:

[0146] According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers;

[0147] Perform feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker;

[0148] Constructing a speech synthesis sample for each speaker based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0149] Model training is performed respectively according to speech synthesis samples of multiple speakers to obtain speech synthesis models of the multiple speakers.

[0150] In an optional implementation, obtaining sample text data of the multiple speakers and acoustic features of the sample audio data of the multiple speakers respectively according to the sample audio data of the multiple speakers includes:

[0151] Generating at least two types of text information corresponding to the sample audio data of each speaker according to the sample audio data of each speaker; the at least two types of text information include: audio text information and voice text information;

[0152] Generate sample text data for each speaker based on at least two types of text information;

[0153] Feature extraction is performed on the sample audio data of each speaker to obtain the acoustic features of the sample audio data of each speaker.

[0154] In an optional implementation, at least two types of text information corresponding to the sample audio data of each speaker are generated according to the sample audio data of each speaker, including:

[0155] Convert the sample audio data of each speaker into text to generate audio text information;

[0156] The spoken text information is generated based on the audio text information and the sample audio data of each speaker.

[0157] In an optional implementation, generating voiced text information based on the audio text information and the sample audio data of each speaker includes:

[0158] Generate pronunciation annotation text information and pronunciation element text information according to the audio text information;

[0159] Extracting pronunciation rhythm according to the audio text information and the sample audio data of each speaker to obtain rhythm annotation text information;

[0160] The pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker; wherein the voiced text information includes: pronunciation annotation text information, pronunciation element text information, rhythm annotation text information and pronunciation element duration information.

[0161] In an optional implementation, the pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker, including:

[0162] Generate initial pronunciation element duration information based on audio text information and sample audio data of each speaker;

[0163] If the initial pronunciation element duration information is a preset null value, a preset alignment tool is used to align the audio text information with the sample audio data of each speaker, and generate the pronunciation element duration information.

[0164] In an optional implementation, the speech synthesis sample includes: a speech synthesis training sample and a speech synthesis test sample;

[0165] According to the speech synthesis samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers, including:

[0166] According to speech synthesis training samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers;

[0167] The method further includes:

[0168] According to the speech synthesis test samples of multiple speakers, the speech synthesis models of the multiple speakers are respectively tested.

[0169] Alternatively, execute the following speech synthesis method:

[0170] According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers;

[0171] Perform feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker;

[0172] Constructing a speech synthesis sample for each speaker based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0173] Model training is performed respectively according to speech synthesis samples of multiple speakers to obtain speech synthesis models of the multiple speakers.

[0174] In the electronic device of this embodiment, the processor executes the speech synthesis model training method or the speech synthesis method. By performing automatic training of the speech synthesis model in a one-click manner, the iteration efficiency of the speech synthesis technology is improved.

[0175] The embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and the computer program is executed by a processor when it is running, and the processor executes the following speech synthesis model training method:

[0176] According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers;

[0177] Perform feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker;

[0178] Constructing a speech synthesis sample for each speaker based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0179] Model training is performed respectively according to speech synthesis samples of multiple speakers to obtain speech synthesis models of the multiple speakers.

[0180] In an optional implementation, obtaining sample text data of the multiple speakers and acoustic features of the sample audio data of the multiple speakers respectively according to the sample audio data of the multiple speakers includes:

[0181] Generating at least two types of text information corresponding to the sample audio data of each speaker according to the sample audio data of each speaker; the at least two types of text information include: audio text information and voice text information;

[0182] Generate sample text data for each speaker based on at least two types of text information;

[0183] Feature extraction is performed on the sample audio data of each speaker to obtain the acoustic features of the sample audio data of each speaker.

[0184] In an optional implementation, at least two types of text information corresponding to the sample audio data of each speaker are generated according to the sample audio data of each speaker, including:

[0185] Convert the sample audio data of each speaker into text to generate audio text information;

[0186] The spoken text information is generated based on the audio text information and the sample audio data of each speaker.

[0187] In an optional implementation, generating voiced text information based on the audio text information and the sample audio data of each speaker includes:

[0188] Generate pronunciation annotation text information and pronunciation element text information according to the audio text information;

[0189] Extracting pronunciation rhythm according to the audio text information and the sample audio data of each speaker to obtain rhythm annotation text information;

[0190] The pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker; wherein the voiced text information includes: pronunciation annotation text information, pronunciation element text information, rhythm annotation text information and pronunciation element duration information.

[0191] In an optional implementation, the pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker, including:

[0192] Generate initial pronunciation element duration information based on audio text information and sample audio data of each speaker;

[0193] If the initial pronunciation element duration information is a preset null value, a preset alignment tool is used to align the audio text information with the sample audio data of each speaker, and generate the pronunciation element duration information.

[0194] In an optional implementation, the speech synthesis sample includes: a speech synthesis training sample and a speech synthesis test sample;

[0195] According to the speech synthesis samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers, including:

[0196] According to speech synthesis training samples of multiple speakers, model training is performed respectively to obtain speech synthesis models of multiple speakers;

[0197] The method further includes:

[0198] According to the speech synthesis test samples of multiple speakers, the speech synthesis models of the multiple speakers are respectively tested.

[0199] Alternatively, execute the following speech synthesis method:

[0200] According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers;

[0201] Perform feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker;

[0202] Constructing a speech synthesis sample for each speaker based on the sample text features of each speaker and the acoustic features of the sample audio data of each speaker;

[0203] Model training is performed respectively according to speech synthesis samples of multiple speakers to obtain speech synthesis models of the multiple speakers.

[0204] In the computer-readable storage medium of this embodiment, the processor executes the speech synthesis model training method or the speech synthesis method. By performing automatic training of the speech synthesis model in a one-click manner, the iteration efficiency of the speech synthesis technology is improved.

[0205] In the embodiment of the present application, the computer program can also execute other machine-readable instructions when run by the processor to execute other methods described in the embodiment. For the specific execution method steps and principles, please refer to the description of the embodiment, which will not be repeated here.

[0206] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0207] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0208] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0209] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0210] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0211] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A speech synthesis model training method, characterized in that: include: According to the sample audio data of the multiple speakers, respectively obtaining the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers; Performing feature extraction on the sample text data of each speaker to obtain the sample text features of each speaker; Constructing a speech synthesis sample of each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker; Model training is performed respectively according to the speech synthesis samples of the multiple speakers to obtain speech synthesis models of the multiple speakers.

2. The method according to claim 1, characterized in that The acquiring, according to the sample audio data of the multiple speakers, the sample text data of the multiple speakers and the acoustic features of the sample audio data of the multiple speakers respectively includes: Generating at least two types of text information corresponding to the sample audio data of each speaker according to the sample audio data of each speaker; the at least two types of text information include: audio text information and voice text information; Generating sample text data of each speaker according to the at least two types of text information; Feature extraction is performed on the sample audio data of each speaker to obtain acoustic features of the sample audio data of each speaker.

3. The method according to claim 2, characterized in that The generating, based on the sample audio data of each speaker, at least two types of text information corresponding to the sample audio data of each speaker includes: Convert the sample audio data of each speaker into text to generate the audio text information; The voiced text information is generated according to the audio text information and the sample audio data of each speaker.

4. The method according to claim 3, characterized in that Generating the voiced text information according to the audio text information and the sample audio data of each speaker includes: Generate pronunciation annotation text information and pronunciation element text information according to the audio text information; Extracting pronunciation rhythm according to the audio text information and the sample audio data of each speaker to obtain rhythm-annotated text information; Pronunciation element duration information is generated according to the audio text information and the sample audio data of each speaker; wherein the voiced text information includes: the pronunciation annotation text information, the pronunciation element text information, the rhythm annotation text information and the pronunciation element duration information.

5. The method according to claim 4, characterized in that The step of generating pronunciation element duration information according to the audio text information and the sample audio data of each speaker includes: Generating initial pronunciation element duration information according to the audio text information and the sample audio data of each speaker; If the initial pronunciation element duration information is a preset null value, a preset alignment tool is used to align the audio text information with the sample audio data of each speaker, and generate the pronunciation element duration information.

6. The method according to claim 1, characterized in that The speech synthesis samples include: speech synthesis training samples and speech synthesis test samples; The method of performing model training respectively according to the speech synthesis samples of the multiple speakers to obtain the speech synthesis models of the multiple speakers includes: Performing model training respectively according to the speech synthesis training samples of the multiple speakers to obtain speech synthesis models of the multiple speakers; The method further comprises: According to the speech synthesis test samples of the multiple speakers, model tests are performed on the speech synthesis models of the multiple speakers respectively.

7. A speech synthesis method, characterized in that: include: Obtain the text to be processed and the identification of the target speaker; Determining a target speech synthesis model from speech synthesis models of multiple speakers according to the identifier of the target speaker, wherein the speech synthesis models of the multiple speakers are trained using the speech synthesis model training method according to any one of claims 1 to 6; Performing feature extraction on the text to be processed to obtain text features of the text to be processed; Processing the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features; According to the acoustic features corresponding to the text features, synthesized speech is obtained.

8. A speech synthesis model training device, characterized in that: include: An acquisition module, used to acquire sample text data of the multiple speakers and acoustic features of the sample audio data of the multiple speakers respectively according to the sample audio data of the multiple speakers; A feature extraction module is used to extract features from the sample text data of each speaker to obtain the sample text features of each speaker; A construction module, used to construct a speech synthesis sample of each speaker according to the sample text features of each speaker and the acoustic features of the sample audio data of each speaker; The training module is used to perform model training respectively according to the speech synthesis samples of the multiple speakers to obtain the speech synthesis models of the multiple speakers.

9. A speech synthesis device, characterized in that: include: An acquisition module, used to acquire the text to be processed and the timbre identifier of the target speaker; A determination module, configured to determine a target speech synthesis model from speech synthesis models of multiple speakers according to the timbre identifier of the target speaker, wherein the speech synthesis models of the multiple speakers are trained using the speech synthesis model according to any one of claims 1 to 6; A feature extraction module, used to extract features of the text to be processed to obtain text features of the text to be processed; A processing module, used for processing the text features of the text to be processed according to the target speech synthesis model to obtain acoustic features corresponding to the text features; The acquisition module is also used to obtain synthesized speech according to the acoustic features corresponding to the text features.

10. An electronic device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the electronic device is running, the processor and the memory communicate via the bus, and the processor executes the machine-readable instructions to execute the speech synthesis model training method described in any one of claims 1 to 6 or the speech synthesis method described in claim 7.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the speech synthesis model training method according to any one of claims 1 to 6 or the speech synthesis method according to claim 7 is executed.

Images