Personalized speech synthesis model creation, speech synthesis and testing method and device

Abstract

The present invention discloses a method for creating a personalized speech synthesis model, a speech synthesis method and a device. The method for creating a personalized speech synthesis model comprises: selecting a speaker of the same category as the user from multiple speakers of a multi-speaker speech synthesis model; training the multi-speaker speech synthesis model according to the training data of the user and the selected speaker of the same category to obtain a personalized speech synthesis model for the user. The present invention can synthesize speech of a specific speaking style of a target speaker, thereby improving the user experience.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method for creating a personalized speech synthesis model, and a speech synthesis and testing method and device. Background Art

[0002] Voice interaction scenarios in artificial intelligence technology require personalized voice synthesis. Personalized voice synthesis is a strong business demand and one of the future trends in the field of voice synthesis.

[0003] In traditional speech synthesis technology, hundreds of hours of training data from hundreds of speakers can be used to build a multi-speaker speech synthesis system based on massive data. Specifically, a multi-speaker speech synthesis model can be used, such as a neural network-based text-to-speech Neural TTS (Text-To-Speech) model. In the training data of this model, the amount of speech data of a single speaker often ranges from a few hours to dozens of hours. A speech synthesis system built using the data of a massive number of speakers can provide a more stable speech synthesis effect. For a multi-speaker Neural TTS model, given any speaker in the training set, the multi-speaker Neural TTS model can be used to synthesize the voice of the speaker.

[0004] For the business demand scenario of personalized speech synthesis, in order to synthesize a personalized speaker's specific style of speech, it is necessary to further learn the characteristics of the personalized speaker based on the above-mentioned multi-speaker Neural TTS model. In order to ensure the learning effect, a large amount of training data of the personalized speaker is required. However, the acquisition of training data requires a high cost investment. Not only is the process cumbersome, but it is also time-consuming, which will seriously affect the user experience. Summary of the invention

[0005] In view of the above problems, the present invention is proposed to provide a method for creating a personalized speech synthesis model, a speech synthesis method, a testing method and a device that overcome the above problems or at least partially solve the above problems.

[0006] In a first aspect, an embodiment of the present invention provides a method for creating a personalized speech synthesis model, comprising:

[0007] Selecting a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model;

[0008] The multi-speaker speech synthesis model is trained according to the training data of the user and the selected speakers of the same type to obtain a personalized speech synthesis model for the user.

[0009] In one embodiment, the user's training data is obtained by: processing the user's data to extract corresponding linguistic features and acoustic features as the user's training data;

[0010] The step of training the multi-speaker speech synthesis model according to the training data of the user and the selected speakers of the same type to obtain a personalized speech synthesis model for the user includes:

[0011] The IDs of the speakers of the same type in the multi-speaker speech synthesis model and the corresponding speaker representations are input into the multi-speaker speech synthesis model, and the multi-speaker speech synthesis model is trained using the user's training data to obtain a personalized speech synthesis model for the user.

[0012] In one embodiment, the user's data includes: voice data and corresponding text;

[0013] The processing of the user's data to extract corresponding linguistic features and acoustic features includes:

[0014] Automatically annotating the user's text through speech synthesis to determine annotation information, the annotation information including: pronunciation annotation and prosody annotation; and determining the speed boundary of the user's speech data through speech recognition and speech activity detection; extracting corresponding linguistic features according to the pronunciation annotation, prosody annotation and speed boundary;

[0015] Extracting acoustic features from the user's voice data.

[0016] In one embodiment, before extracting the acoustic features of the user's voice data, the method further includes:

[0017] The speech data is subjected to preprocessing operations including energy regularization, de-reverberation and energy enhancement.

[0018] In one embodiment, the same category refers to the same category determined according to any one or a combination of the following conditions of the speaker: gender, age, speaking style and speaking environment.

[0019] In one embodiment, in the process of training the multi-speaker speech synthesis model to obtain the personalized speech synthesis model of the user, all model parameters of the multi-speaker speech synthesis model are updated.

[0020] In a second aspect, an embodiment of the present invention provides a method for creating a personalized speech synthesis model, comprising:

[0021] According to a preset scenario, selecting at least one speaker similar to the user from at least one social network of the user corresponding to the scenario;

[0022] Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0023] The multi-speaker speech synthesis model corresponding to the at least one similar speaker is trained according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user.

[0024] In a third aspect, an embodiment of the present invention provides a method for creating a personalized speech synthesis model, including:

[0025] According to the preset priorities of the respective approximate user sets, searching in order of priority for at least one approximate speaker similar to the user in the respective approximate speaker sets;

[0026] Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0027] The multi-speaker speech synthesis model corresponding to the at least one similar speaker is trained according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user.

[0028] In a fourth aspect, an embodiment of the present invention provides a method for creating a personalized speech synthesis model, comprising:

[0029] According to the priorities of the approximate user sets of the user, the approximate speaker sets of each level are pushed to the client of the user in the order of priority;

[0030] receiving at least one approximate speaker selected from the approximate speaker sets of each level returned by the client;

[0031] Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0032] The multi-speaker speech synthesis model corresponding to the at least one similar speaker is trained according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user.

[0033] In a fifth aspect, an embodiment of the present invention provides a personalized speech synthesis method, including:

[0034] Process the text to be synthesized and extract the corresponding linguistic features;

[0035] Inputting the linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model to predict the acoustic features corresponding to the text;

[0036] synthesizing a synthesized speech of the user corresponding to the text according to the acoustic features;

[0037] The personalized speech synthesis model is obtained by using the aforementioned method for creating a personalized speech synthesis model.

[0038] In one embodiment, synthesizing the synthesized speech of the user corresponding to the text according to the acoustic features includes:

[0039] The acoustic features are converted into corresponding speech using a vocoder.

[0040] In a sixth aspect, an embodiment of the present invention provides a method for testing a personalized speech synthesis model, comprising:

[0041] Process the text to be synthesized and extract the corresponding linguistic features;

[0042] Inputting the linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model to predict the acoustic features corresponding to the text;

[0043] synthesizing a synthesized speech of the user corresponding to the text according to the acoustic features;

[0044] Verifying the synthesized speech to determine whether the personalized speech synthesis model is qualified;

[0045] The personalized speech synthesis model is obtained by using the aforementioned method for creating a personalized speech synthesis model.

[0046] In the seventh aspect, the embodiments of the present invention provide a method for creating the aforementioned personalized speech synthesis model, and the application of the aforementioned personalized speech synthesis method and the testing method of the aforementioned personalized speech synthesis model in audio reading, intelligent customer service, voice interaction, voice broadcasting, and machine translation.

[0047] In an eighth aspect, an embodiment of the present invention provides a device for creating a personalized speech synthesis model, including:

[0048] A selection module, for selecting a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model;

[0049] The training module is used to train the multi-speaker speech synthesis model according to the training data of the user and the selected speakers of the same type to obtain a personalized speech synthesis model for the user.

[0050] In one embodiment, the device for creating the personalized speech synthesis model further includes: an extraction module;

[0051] The extraction module is used to process the user's data and extract corresponding linguistic features and acoustic features as the user's training data;

[0052] The training module is specifically used to input the IDs of the speakers of the same type in the multi-speaker speech synthesis model and the corresponding speaker representations into the multi-speaker speech synthesis model, and use the user's training data to train the multi-speaker speech synthesis model to obtain the user's personalized speech synthesis model.

[0053] In a ninth aspect, an embodiment of the present invention provides a device for creating a personalized speech synthesis model, including:

[0054] A first selection module, configured to select, according to a preset scenario, at least one speaker similar to the user from at least one social network of the user corresponding to the scenario;

[0055] A second selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0056] The training module is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

[0057] In a tenth aspect, an embodiment of the present invention provides a device for creating a personalized speech synthesis model, comprising:

[0058] A search module, configured to search for at least one approximate speaker similar to the user in each approximate speaker set in order of priority according to preset priorities of each approximate user set;

[0059] A selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0060] The training module is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

[0061] In an eleventh aspect, an embodiment of the present invention provides a device for creating a personalized speech synthesis model, comprising:

[0062] A push module, configured to push approximate speaker sets of different levels to the client of the user in order of priority according to the priorities of the approximate user sets of the user;

[0063] A receiving module, configured to receive the identifier of the approximate speaker selected from the approximate speaker sets of each level returned by the client;

[0064] A selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0065] The training module is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

[0066] In a twelfth aspect, an embodiment of the present invention provides a personalized speech synthesis device, including:

[0067] The extraction module is used to process the text to be synthesized and extract the corresponding linguistic features;

[0068] A prediction module, used to input the linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized;

[0069] A speech synthesis module, used for synthesizing a synthesized speech corresponding to the user and the text according to the acoustic features;

[0070] The personalized speech synthesis model is obtained by using the aforementioned personalized speech synthesis model creation device.

[0071] In a thirteenth aspect, an embodiment of the present invention provides a testing device for a personalized speech synthesis model, including:

[0072] The extraction module is used to process the text to be synthesized and extract the corresponding linguistic features;

[0073] A prediction module, used to input the linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized;

[0074] A speech synthesis module, used for synthesizing a synthesized speech corresponding to the user and the text according to the acoustic features;

[0075] A verification module, used to verify the synthesized speech and determine whether the personalized speech synthesis model is qualified;

[0076] The personalized speech synthesis model is obtained by using the aforementioned personalized speech synthesis model creation device.

[0077] In the fourteenth aspect, an embodiment of the present invention provides an intelligent voice server, comprising: a memory and a processor; wherein the memory stores a computer program, and when the program is executed by the processor, it can implement the aforementioned method for creating a personalized speech synthesis model, or can implement the aforementioned personalized speech synthesis method, or can implement the aforementioned method for testing a personalized speech synthesis model.

[0078] In the fifteenth aspect, an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, can implement the aforementioned method for creating a personalized speech synthesis model, or can implement the aforementioned personalized speech synthesis method, or can implement the aforementioned method for testing a personalized speech synthesis model.

[0079] The beneficial effects of the above technical solution provided by the embodiment of the present invention include at least:

[0080] The creation method, speech synthesis method, testing method and device of a personalized speech synthesis model provided by the embodiments of the present invention select a similar speaker belonging to the same category as the user from multiple speakers of a multi-speaker speech synthesis model; train the multi-speaker speech synthesis model according to the training data of the user and the selected similar speaker to obtain a personalized speech synthesis model for the user; and use the data of a user with less data (i.e., the target speaker) and the existing multi-speaker speech synthesis model to learn and synthesize the user's speaking voice and style to obtain the user's personalized voice, thereby bringing warmth to the machine and improving the user experience.

[0081] In one embodiment, the same category means trying to find a speaker among multiple speakers that is similar to the target speaker according to gender, age, speaking style and speaking environment, so that the personalized speech synthesis model can better learn the voice of the target speaker.

[0082] In one embodiment, the method for creating a personalized speech synthesis model, the method for speech synthesis, and the device provided by the embodiments of the present invention perform energy regularization, de-reverberation, and energy enhancement preprocessing on the speech data of the target speaker before extracting linguistic features and acoustic features, so that the speech synthesis model has better robustness to environmental noise, reverberation, and varying volumes.

[0083] In one embodiment, the method for creating a personalized speech synthesis model, the method for speech synthesis, and the device for speech synthesis provided by the embodiments of the present invention require updating all model parameters during the process of training the personalized speech synthesis model so as to better learn the voice of the target speaker.

[0084] Another method for creating a personalized speech synthesis model, a speech synthesis method and a device provided by an embodiment of the present invention can also select at least one user similar to the user from at least one social network of the user according to the scenario used, select similar speakers from the at least one similar user, and then use the user's own training data and similar speakers to train the multi-speaker speech synthesis model to obtain the user's personalized speech synthesis model. The embodiment of the present invention can select similar users related to the scenario in the user's social network according to the actual usage scenario, and the user can also independently select the similar users to be imitated. Therefore, more personalized speech model learning can be achieved in a variety of scenarios, further enriching the user experience.

[0085] Other features and advantages of the present invention will be described in the following description, and partly become apparent from the description, or understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained by the structures particularly pointed out in the written description, claims, and drawings.

[0086] The technical solution of the present invention is further described in detail below through the accompanying drawings and embodiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0087] The accompanying drawings are used to provide a further understanding of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings:

[0088] Figure 1 A flowchart of a method for constructing a personalized speech synthesis model provided by an embodiment of the present invention;

[0089] Figure 2 A schematic diagram of the training process of a multi-speaker speech synthesis model provided in an embodiment of the present invention;

[0090] Figure 3 A flowchart of a process for extracting linguistic features provided by an embodiment of the present invention;

[0091] Figure 4 A flowchart of a method for constructing a personalized speech synthesis model in a specific example provided in an embodiment of the present invention;

[0092] Figure 5 A flowchart of extracting linguistic features in a specific example of an embodiment of the present invention;

[0093] Figure 6 A flowchart of extracting acoustic features in a specific example of an embodiment of the present invention;

[0094] Figure 7 A schematic diagram of a training process of a personalized speech synthesis model in a specific example of an embodiment of the present invention;

[0095] Figures 8 to 10 Flowcharts of several other methods for creating personalized speech synthesis models provided by embodiments of the present invention;

[0096] Fig.11 A flowchart of a personalized speech synthesis method provided by an embodiment of the present invention;

[0097] Fig.12 A flowchart of a personalized speech synthesis method provided by an embodiment of the present invention;

[0098] Fig.13 A flowchart of a method for testing a personalized speech synthesis model provided by an embodiment of the present invention;

[0099] Figures 14 to 17 A structural block diagram of a device for creating a personalized speech synthesis model provided by an embodiment of the present invention;

[0100] Fig.18 A structural block diagram of a personalized speech synthesis device provided by an embodiment of the present invention;

[0101] Fig.19 A structural block diagram of a testing device for a personalized speech synthesis model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0102] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0103] The exemplary embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided to enable a more thorough understanding of the present disclosure and to fully convey the scope of the present disclosure to those skilled in the art.

[0104] In order to realize the automatic synthesis of personalized speaker's speech, the embodiment of the present invention provides a method for creating a personalized speech synthesis model, a personalized speech synthesis method, a testing method and a device. For the convenience of description, the user for whom the speech synthesis model is to be generated, that is, the personalized speaker, is called the "target speaker", and the target speaker is not any speaker in the multi-speaker speech synthesis model.

[0105] The specific implementation methods of the above-mentioned schemes are described in detail below in conjunction with the accompanying drawings.

[0106] The method for creating a personalized speech synthesis model provided by the embodiment of the present invention refers to Figure 1 As shown, the following steps are included:

[0107] S11. Selecting a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model;

[0108] S12: training a multi-speaker speech synthesis model according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user.

[0109] In the above step S12, the training data of the above user can be pre-processed in the following manner:

[0110] Process the user's data and extract the corresponding linguistic features and acoustic features as the user's training data;

[0111] Accordingly, in the above step S12, it is necessary to input the IDs and corresponding speaker representations of similar speakers in the preset multi-speaker speech synthesis model into the multi-speaker speech synthesis model, and use the user's training data to train the multi-speaker speech synthesis model, so as to obtain the personalized speech synthesis model of the target speaker.

[0112] The method for creating the personalized speech synthesis model provided by the embodiment of the present invention can utilize the data of a user (target speaker) with a smaller amount of data and an existing multi-speaker speech synthesis model to synthesize the speech of the user (target speaker) with a specific speaking style, thereby obtaining the personalized speech, bringing warmth to the machine and improving the user experience.

[0113] In the method for creating a speech synthesis model provided in the embodiment of the present invention and the speech synthesis method described below, whether it is a multi-speaker speech synthesis model or a personalized speech synthesis model of a user (target speaker), its type can be any neural network-based speech synthesis model (such as a Neural TTS model, etc.), or other similar speech synthesis models such as end-to-end speech synthesis annotation, etc., and the embodiment of the present invention does not limit this.

[0114] First, a multi-speaker speech synthesis model is briefly described. In an embodiment of the present invention, the multi-speaker speech synthesis model can adopt any existing multi-speaker speech synthesis model in the prior art. In the creation of the multi-speaker speech synthesis model, the model is trained using training set data of multiple speakers, and the multiple speakers are pre-set speakers or certain speakers, such as Zhang San, Li Si and Wang Wu, etc. Each person has a corresponding ID (number) in this model.

[0115] The multi-speaker training set data includes the training data of each speaker, and each person's training data may include the speaker's speech data and corresponding text, as well as linguistic features and acoustic features extracted from the speech data and text.

[0116] Different personalized speech synthesis models can be targeted at different personalized individuals, and the multi-speaking speech synthesis model is the basis of the personalized speech synthesis model. In order to ensure the learning accuracy of the model, the extraction of linguistic features and acoustic features can be achieved through a series of means, and a variety of methods are also used in specific implementations, such as manual annotation or computer-assisted manual annotation. The embodiments of the present invention do not limit this.

[0117] The training process of a multi-speaker speech synthesis model can be referred to Figure 2 As shown in the figure, when training a multi-speaker speech synthesis model, an ID is set for each speaker. Suppose the training set data contains data of three speakers with IDs 1, 2, and 3. During training, the input data is the linguistic features of the three speakers with IDs 1, 2, and 3 and their corresponding IDs and speaker embeddings. Figure 2As shown in the figure, the multi-speaker speech synthesis model includes encoder, attention mechanism and decoder. The output is the acoustic features of the three people. The training process can be implemented by a neural network training method such as the back propagation algorithm (BP, BackPropagation) algorithm. The principle of the BP algorithm mainly consists of two links, namely, excitation propagation and weight update, which are repeated and iterated until the network's response to the input reaches the predetermined target range. The learning process of the BP algorithm consists of a forward propagation process and a back propagation process. In the forward propagation process, the input information passes through the input layer and the hidden layer, and is processed layer by layer and transmitted to the output layer. If the expected output value is not obtained in the output layer, the sum of the squares of the error between the output and the expected value is taken as the objective function, and the back propagation is turned to. The partial derivatives of the objective function to each neuron weight are obtained layer by layer, and the gradient of the objective function to the weight vector is formed as the basis for modifying the weight. The learning of the network is completed in the weight modification process. When the error reaches the expected value, the network learning ends. By learning using training set data, the relationship between linguistic features and acoustic features can be learned.

[0118] Taking the multi-speaker speech synthesis model with IDs 1, 2 and 3 as an example, when training the multi-speaker speech synthesis model, the input is the linguistic features of the three speakers with IDs 1, 2 and 3 and the corresponding speaker IDs, plus the speaker representations of the three speakers. The speaker representation contains the representation vectors of the three speakers ID1, ID2 and ID3. After the training is completed, if the ID of a speaker in the training set is input, and any linguistic feature is input, the corresponding voice of the speaker with ID 1 can be predicted.

[0119] Speaker Embedding contains a set of feature vectors, the number of which is equal to the total number of speakers in the training set of multiple speakers. Mathematically, it is represented by an N*M matrix (N is the number of speakers). The feature vector of each speaker is abstracted and quantified from a series of features of the speaker.

[0120] The multi-speaker speech synthesis model also has the following characteristics: if the multi-speaker speech synthesis model does not have a speaker ID and a speaker representation, the model can only output a standard voice based on the text, but if there is an ID and a speaker representation of a specific speaker (such as Zhang San, ID is 1) in the training set data of the multi-speaker speech synthesis model, the voice of the specific speaker can be output, that is, the multi-speaker speech synthesis model can be controlled to output the voice of any person in the training set, but for speakers not in the training set data, the multi-speaker speech synthesis model cannot output the corresponding voice, in other words, the input ID must be the ID of a speaker in the training set. The embodiment of the present invention utilizes the above characteristics, so that the user (target speaker) can "impersonate" a speaker in the trained multi-speaker speech synthesis model, so that the existing multi-speaker speech synthesis model can be used to implement the training of the personalized user (target speaker) speech synthesis model.

[0121] In one embodiment, in the above step S11, in the step of processing the user (target speaker) data to extract the corresponding linguistic features and acoustic features, the user (target speaker) data includes: speech data and corresponding text.

[0122] The process of extracting linguistic features, refer to Figure 3 As shown, this can be achieved, for example, by the following steps:

[0123] S31, automatically annotating the user's text through speech synthesis to determine annotation information, where the annotation information includes: pronunciation annotation and rhythm annotation;

[0124] For example, the text is processed through the TTS front end to perform pronunciation and rhythm annotation.

[0125] S32, determining the speed boundary of the user's voice data through voice recognition and voice activity detection;

[0126] S33. Extract corresponding linguistic features according to pronunciation annotation, prosody annotation and speed boundary.

[0127] Linguistic features refer to linguistic features extracted based on pronunciation annotation and prosodic annotation, such as phoneme sequence, tone, boundary information and pause.

[0128] For example, the pronunciation marking is to mark the text with pinyin (including tone), for example, "wo3" is marked as "wo3", and the number 3 indicates that the tone is the third tone.

[0129] Prosody annotation, such as marking pause marks, for example, in the sentence "I am #3China#1.", "#3" indicates a long pause, and "#1" indicates a short pause.

[0130] The phoneme boundaries are determined for the speech data. Specifically, for example, the start time and end time of each phoneme can be determined through technical means such as automatic speech recognition (ASR), voice activity detection (VAD, Voice Activity Detection, the function of which is to determine the silent part of the speech), etc.

[0131] In the embodiment of the present invention, the acoustic features are acoustic features extracted based on speech, such as linear spectrum, Mel-scale Frequency Cepstral Coefficients (MFCC) and Fbank.

[0132] MFCC is the cepstrum parameter extracted in the Mel scale frequency domain. According to the research on the human ear hearing mechanism, the human ear has different auditory sensitivities to sound waves of different frequencies. The speech signal from 200Hz to 5000Hz has the greatest impact on the clarity of speech. When two sounds of different loudness act on the human ear, the presence of the higher frequency component will affect the perception of the lower frequency component, making it difficult to detect. This phenomenon is called the masking effect. Since the distance of the lower frequency sound traveling on the basilar membrane of the inner cochlea is greater than that of the higher frequency sound, generally speaking, bass is easy to mask treble, while treble is difficult to mask bass. The critical bandwidth of sound masking at low frequencies is smaller than that of high frequencies. Therefore, people arrange a group of bandpass filters from dense to sparse according to the size of the critical bandwidth in the frequency band from low frequency to high frequency to filter the input signal. The signal energy output by each bandpass filter is taken as the basic feature of the signal, and this feature can be used as the input feature of speech after further processing. Since this feature does not depend on the nature of the signal, does not make any assumptions or restrictions on the input signal, and utilizes the research results of the auditory model, this parameter has better robustness than the LPCC based on the vocal tract model, is more in line with the auditory characteristics of the human ear, and still has good recognition performance when the signal-to-noise ratio is reduced.

[0133] The MFCC extraction process includes: pre-emphasis, framing, windowing, FFT (Fast Fourier Transform), triangular bandpass filter filtering, calculating the logarithmic energy of each filter group output, and then obtaining the MFCC coefficients through discrete cosine transform (DCT).

[0134] The extraction of MFCC coefficients is obtained by performing discrete cosine transform on the basis of Fbank. Therefore, the process of extracting Fbank is consistent with the first few steps of extracting MFCC coefficients.

[0135] The extraction of the linear spectrum is achieved in the following manner: a sliding window Fourier transform is performed on the voice data signal, and the linear spectrum of the voice signal is obtained after the sliding window Fourier transform processing.

[0136] The embodiment of the present invention does not limit the specific features used for the linguistic features and acoustic features, nor does it limit the specific extraction method of the features, and the extraction means in the prior art can be used to achieve the feature.

[0137] Because the voice collection method of the user's (i.e., the target speaker) voice data is relatively random, for example, using a mobile phone to speak in an environment with a cluttered background, the user's voice may contain varying degrees of environmental noise and reverberation, and its volume may also vary. In order to achieve better training effects even in a poor recording environment, preferably, in an embodiment of the present invention, before extracting the acoustic features of the user's voice data, the following steps may also be performed: preprocessing the voice data including energy regularization, de-reverberation, and energy enhancement.

[0138] Specifically, the energy regularization step is to regularize the energy of the same batch of data to a specific energy distribution. The de-reverberation step is to eliminate the reverberation in the speech; the energy enhancement step is to enhance the speech signal and reduce the noise.

[0139] In theory, the more speech data a user (i.e., the target speaker) has and the longer the speech is, the more beneficial it is for training a personalized speech synthesis model. However, in practice, too much data will increase the cost of recording and speech processing (such as annotation, extraction of linguistic and acoustic features) and reduce the user experience. Therefore, in an embodiment of the present invention, the user (i.e., the target speaker) can provide about 10 to 100 sentences of speech data to complete the creation of a personalized speech synthesis model, which greatly improves the user experience while ensuring the accuracy of the model.

[0140] In one embodiment, the embodiment of the present invention needs to select a speaker of the same category as the user (ie, the target speaker) from among multiple speakers in the training set of the multi-speaker speech synthesis model.

[0141] The same category refers to the same category determined according to any one or a combination of the following conditions of the speaker: gender, age, speaking style and speaking environment, etc. Speakers of the same category are selected by comprehensively considering one or more of the conditions.

[0142] For example, you can directly select a speaker of the same gender, or you can select a speaker of the same gender and age group, and so on. The general principle is to select a speaker that is closer to the user (ie, the target speaker).

[0143] If the selection is simple, a single condition can be considered, for example, speakers of the same gender can be selected. Since there are large differences between male and female voices, speakers of the same gender can be simply selected.

[0144] Trying to find a speaker among multiple speakers that is similar to the user (target speaker) can enable the personalized speech synthesis model to better learn the voice of the user (target speaker).

[0145] The following briefly describes the method for creating a personalized speech synthesis model provided by an embodiment of the present invention using a specific example.

[0146] Reference Figure 4 In the flowchart shown, it is assumed that the training set data of multiple speakers of the multi-speaker speech synthesis model includes speakers A, B, C, D and E, whose IDs in the multi-speaker speech synthesis model are ID1, ID2, ID3, ID4 and ID5 respectively. The multi-speaker Neural TTS model is trained using the training data of these speakers to obtain a trained multi-speaker Neural TTS model.

[0147] Currently, there is a personalized speaker, speaker F. After speech synthesis, automatic annotation and speech data preprocessing are performed on the personalized data of speaker F, namely, speech data and text, the corresponding linguistic features and acoustic features are extracted.

[0148] Figure 5 The figure shows the process of how to extract linguistic features from text and speech. For example, the pronunciation annotations and prosody annotations in the text are first extracted through the TTS front end, and the speech is processed by ASR and VAD to obtain phoneme boundaries. The linguistic features are then extracted based on the pronunciation annotations, prosody annotations and phoneme boundaries.

[0149] Figure 6 Shown is the process of extracting the corresponding acoustic features after preprocessing the speech (energy regularization, de-reverberation and energy enhancement).

[0150] From speaker A, speaker B, speaker C, speaker D, and speaker E, a speaker of the same gender as speaker F, for example, speaker E, is selected.

[0151] Reference Figure 7 As shown, the ID of speaker E, i.e., ID5, and the speaker representation of speaker E are input into the Attention of the multi-speaker Neural TTS model, and the linguistic features and acoustic features of the user (target speaker), i.e., speaker F, are used to train the multi-speaker Neural TTS model, so as to obtain a personalized speech synthesis model for speaker F.

[0152] The inventor of the present invention has verified through experiments that the method for creating a personalized speech synthesis model provided by the embodiment of the present invention can well simulate the speaker's voice when the user (i.e., the target speaker F) provides fewer sentences, for example, even if there are only 10 sentences, so that the user can use it conveniently and will not be limited by the recording environment and the recording equipment used. In addition, the amount of data of the user (target speaker F) is small, and the time cost of recording and annotation is greatly reduced, which also greatly speeds up the training process of the speech synthesis model and the speed of speech synthesis.

[0153] The embodiment of the present invention also provides a method for creating a personalized speech synthesis model, referring to Figure 8 As shown, at least the following steps are included:

[0154] S81, selecting at least one speaker similar to the user from at least one social network of the user corresponding to the scenario according to a preset scenario;

[0155] S82, selecting a speaker of the same category as the user from at least one similar speaker;

[0156] S83. According to the user's training data and the selected similar speakers, a multi-speaker speech synthesis model corresponding to at least one similar speaker is trained to obtain a personalized speech synthesis model for the user.

[0157] In the above method, there can be many scenarios. For example, if the client APP corresponds to a social network scenario, the server can select users similar to the user from at least one social network of the user according to the applicable scenario of the client APP used by the user, such as a social APP, a voice imitation APP, or an online game APP, according to the specific scenario. For example, from the family members, relatives, friends, colleagues, classmates and other groups in the user's circle of friends, select users who have one or some common points with the user, such as the same family, the same school, or the same workplace, etc., select at least one approximate speaker similar to the user from these social networks, and then select similar speakers belonging to the same category as the user based on these at least one speaker, and use the user's training data to train a multi-speaker speech synthesis model of these at least one approximate speakers to obtain a personalized speech synthesis model for the user.

[0158] For another example, this scenario can also be a scenario set by the user on the client, such as a home scenario, a work scenario, a leisure scenario, etc. In different scenarios, the user may want to use different voices and / or different language expressions, or in certain specific occasions, want to use specific voices and / or language expressions. In this way, the user can independently choose the corresponding personalized speech synthesis model that needs to learn the voices and / or expression styles of those similar users. The user can use the client to select similar users from multiple social networks, or directly choose to learn the voice of one or more specific characters in specific occasions, and search for users of the same category based on these similar users, so as to achieve the purpose of imitating the voice and / or expression style of the similar user.

[0159] The embodiment of the present invention also provides a method for creating a personalized speech synthesis model, referring to Fig. 9 As shown, the following steps are included:

[0160] S91, searching for at least one approximate speaker similar to the user in each approximate speaker set in order of priority according to preset priorities of each approximate user set;

[0161] S92. Selecting a speaker of the same type as the user from at least one similar speaker;

[0162] S93. According to the user's training data and the selected similar speakers, a multi-speaker speech synthesis model corresponding to at least one similar speaker is trained to obtain a personalized speech synthesis model for the user.

[0163] In the above steps S91 to S93, there may be multiple sets of similar users selected in different ways, such as the set of similar users selected from the social network mentioned above. According to the preset priority, the server can search for at least one similar speaker similar to the user from each similar speaker set in turn. For example, sets with different priorities include: a set of family members, a set of friends, a set of classmates, a set of colleagues, a set of residential communities, and a set as large as an administrative area such as a district, a city, a province, etc. The priority levels are from high to low. At this time, when selecting similar speakers, they can be selected in order from high to low priority until a sufficient number of similar users are selected. If the selection of all similar speakers is met in a higher set, there is no need to search in the set of the next level.

[0164] The selection of approximate speakers or approximate users can also be completely left to the user (target speaker) to choose. For example, the server pushes the corresponding approximate user set, and the client selects it. After completion, it returns to the server to obtain training data and train a personalized speech synthesis model.

[0165] Then, based on the at least one speaker, similar speakers belonging to the same category as the user are selected, and the multi-speaker speech synthesis model of the at least one similar speaker is trained using the user's training data to obtain a personalized speech synthesis model for the user.

[0166] For the creation method of the corresponding personalized speech synthesis model, refer to Fig.10 As shown, the following steps are included:

[0167] S101, according to the priorities of the approximate user sets of the user, push the approximate speaker sets of different levels to the client of the user in the order of priority;

[0168] S102, receiving at least one approximate speaker selected from the approximate speaker sets of each level returned by the client;

[0169] S103, selecting a speaker of the same type as the user from at least one similar speaker;

[0170] S104: Train a multi-speaker speech synthesis model corresponding to at least one similar speaker according to the user's training data and the selected similar speakers to obtain a personalized speech synthesis model for the user.

[0171] The set of similar speakers at the above levels includes one or more of the following:

[0172] At least one user set of the user's social network; as mentioned above, the user's social network may be friends of multiple social groups selected by the user, etc., which will not be elaborated here.

[0173] The users belong to at least one set of users in the same geographical area, such as users in the same district, city, or even the same area of ​​the same province. This scenario can be applicable when a personalized speech synthesis model needs to learn the dialect and pronunciation characteristics of a specific region.

[0174] The user selects at least one user set according to his / her own preferences; for example, when the user wants to imitate the voice and / or language style of a specific person, he / she can select one or more preferred users as approximate users (approximate speakers) according to his / her own preferences.

[0175] In the specific implementation, the above S101 to S104 can be realized by a server or a client with certain computing capabilities, and the user selects an approximate speaker from multiple possible sets, such as a favorite star, to merge the voice characteristics of the user and the star in a certain proportion to produce a rich voice experience. The server or client selects a speaker of the same type as the user based on the identifiers of these approximate speakers, and then trains the multi-speaker speech synthesis models corresponding to these approximate speakers based on the training data of the same speakers and users, and generates a personalized speech synthesis model corresponding to the user.

[0176] The embodiment of the present invention also provides a personalized speech synthesis method based on the method for creating the personalized speech synthesis model.

[0177] Reference Fig.11 As shown, the personalized speech synthesis method provided by the embodiment of the present invention includes the following steps:

[0178] S111, processing the text requiring speech synthesis and extracting corresponding linguistic features;

[0179] S112, inputting the linguistic features and the ID of the same speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predicting the acoustic features corresponding to the text;

[0180] S113. Synthesize the synthetic speech corresponding to the user and the text according to the acoustic features.

[0181] The above-mentioned personalized speech synthesis model is obtained by using the above-mentioned method for creating a personalized speech synthesis model.

[0182] by Fig.12 Taking the flowchart shown as an example, assuming that the user (i.e., target speaker F) needs to synthesize speech about a short text, and the speaker of the same type selected by speaker F during the training of the personalized Neural TTS model is speaker E, the TTS front end is first used to extract the linguistic features of the text to be synthesized, and then the extracted linguistic features and speaker E's ID, i.e., ID5, are input into the personalized Neural TTS model, and the corresponding acoustic features can be output, and then the acoustic features are converted into the speech of speaker F through the vocoder. The text that needs speech synthesis can be arbitrarily specified according to the needs.

[0183] Reference Fig.13As shown, the testing method for the personalized speech synthesis model provided in the embodiment of the present invention, S131~S133 are similar to the above-mentioned speech synthesis method, except that the final step S134 test further needs to verify the speech result output by the model to determine whether the personalized speech synthesis model is qualified. If the test fails, the training process of the model can be adjusted according to the result feedback. The specific implementation process will not be repeated here.

[0184] The training method of the personalized speech synthesis model and the personalized speech synthesis method provided in the embodiments of the present invention can be widely used in various artificial intelligence scenarios, such as in application scenarios such as audio reading, intelligent customer service, voice interaction, voice broadcasting, and machine translation.

[0185] For example, when applied to voice assistants, intelligent customer service and other products, the user inputs a piece of self-recorded voice in advance. Given any text content, the system can output realistic voice, thereby realizing intelligent interaction, voice broadcast and other scenarios.

[0186] Based on the same inventive concept, the embodiments of the present invention also provide a personalized speech synthesis model creation device, a personalized speech synthesis device, a personalized speech synthesis model testing device and a server. Since the principles of the problems solved by these devices and servers are similar to the aforementioned personalized speech synthesis model creation method, personalized speech synthesis method and personalized speech synthesis model testing method, the implementation of the device and server can refer to the implementation of the aforementioned method, and the repeated parts will not be repeated.

[0187] An embodiment of the present invention provides a device for creating a personalized speech synthesis model, referring to Fig.14 As shown, including:

[0188] A selection module 141, configured to select a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model;

[0189] The training module 142 is used to train the multi-speaker speech synthesis model according to the training data of the user and the selected speakers of the same type to obtain a personalized speech synthesis model for the user.

[0190] Continue to refer to Fig.14 As shown, the personalized speech synthesis model further includes: an extraction module 143;

[0191] The extraction module 143 is used to process the user's data and extract corresponding linguistic features and acoustic features as the user's training data;

[0192] The training module 142 is specifically configured to input the IDs of the speakers of the same type in the multi-speaker speech synthesis model and the corresponding speaker representations into the multi-speaker speech synthesis model, and train the multi-speaker speech synthesis model using the user's training data to obtain a personalized speech synthesis model for the target speaker.

[0193] In one embodiment, the data of the target speaker includes: speech data and corresponding text; accordingly, the above-mentioned extraction module 143 is further used to automatically annotate the text of the target speaker through speech synthesis to determine annotation information, and the annotation information includes: pronunciation annotation and prosody annotation; and determine the speed boundary of the speech data of the target speaker through speech recognition and speech activity detection; extract corresponding linguistic features according to the pronunciation annotation, prosody annotation and speed boundary; extract acoustic features of the speech data of the target speaker.

[0194] In one embodiment, the extraction module 143 is further configured to perform preprocessing operations including energy regularization, de-reverberation and energy enhancement on the speech data before extracting acoustic features from the speech data of the target speaker.

[0195] In one embodiment, the same category refers to the same category determined according to any one or a combination of the following conditions of the speaker: gender, age, speaking style and speaking environment.

[0196] In one embodiment, the training module 142 is further used to train the multi-speaker speech synthesis model, and in the process of obtaining the personalized speech synthesis model of the target speaker, update all model parameters of the multi-speaker speech synthesis model.

[0197] The embodiment of the present invention provides a device for creating a personalized speech synthesis model, referring to Fig.15 As shown, including:

[0198] A first selection module 151, configured to select, according to a preset scenario, at least one speaker similar to the user from at least one social network of the user corresponding to the scenario;

[0199] A second selection module 152, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0200] The training module 153 is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

[0201] The embodiment of the present invention provides a device for creating a personalized speech synthesis model, referring to Fig.16 As shown, including:

[0202] A search module 161 is used to search for at least one approximate speaker similar to the user in each approximate speaker set in order of priority according to the preset priorities of each approximate user set;

[0203] A selection module 162, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0204] The training module 163 is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain a personalized speech synthesis model for the user.

[0205] The embodiment of the present invention provides a device for creating a personalized speech synthesis model, referring to Fig.17 As shown, including:

[0206] A push module 171 is used to push approximate speaker sets of different levels to the client of the user in the order of priority according to the priorities of the approximate user sets of the user;

[0207] A receiving module 172, configured to receive the identifier of the approximate speaker selected from the approximate speaker sets of each level returned by the client;

[0208] A selection module 173, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker;

[0209] The training module 174 is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

[0210] The embodiment of the present invention also provides a personalized speech synthesis device, referring to Fig.18 As shown, including:

[0211] An extraction module 181 is used to process the text to be synthesized and extract corresponding linguistic features;

[0212] A prediction module 182, configured to input the linguistic features and the ID of a similar speaker corresponding to the target speaker during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized;

[0213] A speech synthesis module 183, configured to synthesize a synthesized speech corresponding to the target speaker and the text according to the acoustic features;

[0214] The above-mentioned personalized speech synthesis model is obtained by using the above-mentioned personalized speech synthesis model creation device.

[0215] The embodiment of the present invention also provides a testing device for a personalized speech synthesis model, referring to Fig.19 As shown, including:

[0216] An extraction module 191 is used to process the text to be synthesized and extract corresponding linguistic features;

[0217] A prediction module 192, configured to input the linguistic features and the ID of a similar speaker corresponding to the target speaker during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized;

[0218] A speech synthesis module 193, configured to synthesize a synthesized speech corresponding to the target speaker and the text according to the acoustic features;

[0219] A verification module 194, used to verify the synthesized speech and determine whether the personalized speech synthesis model is qualified;

[0220] Similarly, the above-mentioned personalized speech synthesis model is also obtained by using the above-mentioned personalized speech synthesis model creation device.

[0221] An embodiment of the present invention also provides an intelligent voice server, comprising: a memory and a processor; wherein the memory stores a computer program, and when the program is executed by the processor, it can implement the method for creating the aforementioned personalized speech synthesis model, or can implement the aforementioned personalized speech synthesis method, or can implement the aforementioned testing method of the personalized speech synthesis model.

[0222] An embodiment of the present invention also provides a non-temporary computer-readable storage medium. When the instructions in the storage medium are executed by a processor, it can execute the method for creating the aforementioned personalized speech synthesis model, or can implement the aforementioned personalized speech synthesis method, or can implement the testing method of the aforementioned personalized speech synthesis model.

[0223] The personalized speech synthesis model creation method, speech synthesis method, personalized speech synthesis model testing method and device provided by the embodiments of the present invention require updating all model parameters during the process of training the personalized speech synthesis model so as to better learn the voice of the target speaker.

[0224] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage and optical storage, etc.) containing computer-usable program code.

[0225] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0226] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0227] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0228] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for creating a personalized speech synthesis model, characterized in that: include: Selecting a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model; The multi-speaker speech synthesis model is trained according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user, wherein when the multi-speaker speech synthesis model is trained, an ID is set for each speaker, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The method further includes: obtaining scenarios for personalized speech synthesis, wherein the language expressions required by the user in different scenarios are different; and selecting the same type of speakers from at least one social network corresponding to the scenario.

2. The method according to claim 1, characterized in that The user's training data is obtained by: processing the user's data to extract corresponding linguistic features and acoustic features as the user's training data; The step of training the multi-speaker speech synthesis model according to the training data of the user and the selected speakers of the same type to obtain a personalized speech synthesis model for the user includes: The IDs of the speakers of the same type in the multi-speaker speech synthesis model and the corresponding speaker representations are input into the multi-speaker speech synthesis model, and the multi-speaker speech synthesis model is trained using the user's training data to obtain a personalized speech synthesis model for the user.

3. The method according to claim 2, characterized in that The user's data includes: voice data and corresponding text; The processing of the user's data to extract corresponding linguistic features and acoustic features includes: Automatically annotating the user's text through speech synthesis to determine annotation information, the annotation information including: pronunciation annotation and prosody annotation; and determining the speed boundary of the user's speech data through speech recognition and speech activity detection; extracting corresponding linguistic features according to the pronunciation annotation, prosody annotation and speed boundary; Extracting acoustic features from the user's voice data.

4. The method according to claim 2, characterized in that Before extracting the acoustic features of the user's voice data, the method further includes: The speech data is subjected to preprocessing operations including energy regularization, de-reverberation and energy enhancement.

5. The method according to any one of claims 1 to 4, characterized in that: The same category refers to the same category determined according to any one or a combination of the following conditions of the speaker: gender, age, speaking style and speaking environment.

6. The method according to any one of claims 1 to 4, characterized in that: In the process of training the multi-speaker speech synthesis model to obtain the personalized speech synthesis model of the user, all model parameters of the multi-speaker speech synthesis model are updated.

7. A method for creating a personalized speech synthesis model, characterized in that: include: According to a preset scenario, selecting at least one speaker similar to the user from at least one social network of the user corresponding to the scenario, wherein the language expression required by the user in different scenarios is different; Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker; According to the training data of the user and the selected similar speakers, a multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained to obtain a personalized speech synthesis model for the user, wherein when the multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained, an ID is set for each of the approximate speakers, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model.

8. A method for creating a personalized speech synthesis model, characterized in that: include: According to the preset priorities of the respective approximate user sets, searching in order of priority for at least one approximate speaker similar to the user in the respective approximate speaker sets; Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker; According to the training data of the user and the selected similar speakers, the multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained to obtain a personalized speech synthesis model for the user, wherein when the multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained, an ID is set for each of the approximate speakers, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The method further includes: obtaining a scenario for performing personalized speech synthesis, wherein the language expression required by the user in different scenarios is different; and selecting at least one approximate speaker from at least one social network corresponding to the scenario.

9. A method for creating a personalized speech synthesis model, characterized in that: include: According to the priorities of the approximate user sets of the user, the approximate speaker sets of each level are pushed to the client of the user in the order of priority; receiving at least one approximate speaker selected from the approximate speaker sets of each level returned by the client; Selecting a similar speaker belonging to the same category as the user from the at least one similar speaker; According to the training data of the user and the selected similar speakers, the multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained to obtain a personalized speech synthesis model for the user, wherein when the multi-speaker speech synthesis model corresponding to the at least one approximate speaker is trained, an ID is set for each of the approximate speakers, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The method further includes: obtaining a scenario for performing personalized speech synthesis, wherein the language expression required by the user in different scenarios is different; and selecting at least one approximate speaker from at least one social network corresponding to the scenario.

10. A personalized speech synthesis method, characterized in that: include: Process the text to be synthesized and extract the corresponding linguistic features; Inputting the linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predicting the acoustic features corresponding to the text, wherein the similar speaker is selected from at least one social network corresponding to a scenario for personalized speech synthesis, and the language expression required by the user in different scenarios is different; synthesizing a synthesized speech of the user corresponding to the text according to the acoustic features; The personalized speech synthesis model is obtained by using the method for creating a personalized speech synthesis model as described in any one of claims 1-8.

11. The method according to claim 10, characterized in that Synthesizing, according to the acoustic features, a synthesized speech of the user corresponding to the text, comprising: The acoustic features are converted into corresponding speech using a vocoder.

12. A method for testing a personalized speech synthesis model, characterized in that: include: Process the text to be synthesized and extract the corresponding linguistic features; The linguistic features and the ID of a similar speaker corresponding to the user during the training process of the personalized speech synthesis model are input into the personalized speech synthesis model to predict the acoustic features corresponding to the text, wherein the similar speaker is selected from at least one social network corresponding to a scenario for personalized speech synthesis, and the language expression required by the user in different scenarios is different; synthesizing a synthesized speech of the user corresponding to the text according to the acoustic features; Verifying the synthesized speech to determine whether the personalized speech synthesis model is qualified; The personalized speech synthesis model is obtained by using the method for creating a personalized speech synthesis model as described in any one of claims 1-9.

13. Application of the method for creating a personalized speech synthesis model according to any one of claims 1 to 9, the personalized speech synthesis method according to claim 10 or 11, and the testing method of the personalized speech synthesis model in audio reading, intelligent customer service, voice interaction, voice broadcasting, and machine translation.

14. A device for creating a personalized speech synthesis model, characterized in that: include: A selection module, for selecting a speaker of the same category as the user from among multiple speakers of the multi-speaker speech synthesis model; A training module, used for training the multi-speaker speech synthesis model according to the training data of the user and the selected similar speakers to obtain a personalized speech synthesis model for the user, wherein when training the multi-speaker speech synthesis model, an ID is set for each speaker, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The device is also used to perform the following steps: obtaining a scenario for personalized speech synthesis, wherein the language expression required by the user in different scenarios is different; and selecting the same type of speaker from at least one social network corresponding to the scenario.

15. The device according to claim 14, characterized in that Also includes: Extraction module; The extraction module is used to process the user's data and extract corresponding linguistic features and acoustic features as the user's training data; The training module is specifically used to input the IDs of the speakers of the same type in the multi-speaker speech synthesis model and the corresponding speaker representations into the multi-speaker speech synthesis model, and use the user's training data to train the multi-speaker speech synthesis model to obtain the user's personalized speech synthesis model.

16. A device for creating a personalized speech synthesis model, characterized in that: include: A first selection module is used to select at least one speaker similar to the user from at least one social network of the user corresponding to the scenario according to a preset scenario, wherein the language expression required by the user in different scenarios is different; A second selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker; The training module is used to train the multi-speaker speech synthesis model corresponding to the at least one similar speaker according to the training data of the user and the selected similar speakers, so as to obtain the personalized speech synthesis model of the user.

17. A device for creating a personalized speech synthesis model, characterized in that: include: A search module, configured to search for at least one approximate speaker similar to the user in each approximate speaker set in order of priority according to preset priorities of each approximate user set; A selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker; A training module, configured to train a multi-speaker speech synthesis model corresponding to the at least one approximate speaker according to the training data of the user and the selected similar speaker, so as to obtain a personalized speech synthesis model for the user, wherein when training the multi-speaker speech synthesis model corresponding to the at least one approximate speaker, an ID is set for each of the approximate speakers, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The device is also used to perform the following steps: obtaining a scenario for personalized speech synthesis, wherein the language expression required by the user in different scenarios is different; and selecting at least one approximate speaker from at least one social network corresponding to the scenario.

18. A device for creating a personalized speech synthesis model, characterized in that: include: A push module, configured to push approximate speaker sets of different levels to the client of the user in order of priority according to the priorities of the approximate user sets of the user; A receiving module, configured to receive the identifier of the approximate speaker selected from the approximate speaker sets of each level returned by the client; A selection module, configured to select a similar speaker belonging to the same category as the user from the at least one similar speaker; A training module, configured to train a multi-speaker speech synthesis model corresponding to the at least one approximate speaker according to the training data of the user and the selected similar speaker, so as to obtain a personalized speech synthesis model for the user, wherein when training the multi-speaker speech synthesis model corresponding to the at least one approximate speaker, an ID is set for each of the approximate speakers, and the ID and the speaker representation corresponding to the ID are input into the multi-speaker speech synthesis model; The device is also used to perform the following steps: obtaining a scenario for personalized speech synthesis, wherein the language expression required by the user in different scenarios is different; and selecting at least one approximate speaker from at least one social network corresponding to the scenario.

19. A personalized speech synthesis device, characterized in that: include: The extraction module is used to process the text to be synthesized and extract the corresponding linguistic features; A prediction module, used to input the linguistic features and the ID of a similar speaker corresponding to the user in the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized, wherein the similar speaker is selected from at least one social network corresponding to the scenario for personalized speech synthesis, and the language expression required by the user in different scenarios is different; A speech synthesis module, used for synthesizing a synthesized speech corresponding to the user and the text according to the acoustic features; The personalized speech synthesis model is obtained using the personalized speech synthesis model creation device as described in any one of claims 14-18.

20. A testing device for a personalized speech synthesis model, characterized in that: include: The extraction module is used to process the text to be synthesized and extract the corresponding linguistic features; A prediction module, used to input the linguistic features and the ID of a similar speaker corresponding to the user in the training process of the personalized speech synthesis model into the personalized speech synthesis model, and predict the acoustic features corresponding to the text to be speech synthesized, wherein the similar speaker is selected from at least one social network corresponding to the scenario for personalized speech synthesis, and the language expression required by the user in different scenarios is different; A speech synthesis module, used for synthesizing a synthesized speech corresponding to the user and the text according to the acoustic features; A verification module, used to verify the synthesized speech and determine whether the personalized speech synthesis model is qualified; The personalized speech synthesis model is obtained using the personalized speech synthesis model creation device as described in any one of claims 14-18.

21. An intelligent voice server, characterized in that: include: A memory and a processor; wherein the memory stores a computer program, which, when executed by the processor, can implement a method for creating a personalized speech synthesis model as described in any one of claims 1 to 9, or can implement a personalized speech synthesis method as described in claim 10 or 11, or can implement a method for testing a personalized speech synthesis model as described in claim 12 or 13.

22. A computer-readable storage medium having computer instructions stored thereon, characterized in that: When this instruction is executed by the processor, it can implement a method for creating a personalized speech synthesis model as described in any one of items 1-9, or can implement a personalized speech synthesis method as described in claim 10 or 11, or can implement a method for testing a personalized speech synthesis model as described in claim 12 or 13.

Images