A method, apparatus, storage medium and electronic device for audio synthesis

Abstract

Some embodiments of the present application provide a method, device, storage medium and electronic equipment for audio synthesis, the method comprising: obtaining a timbre feature set corresponding to speech audio of a target object, wherein the timbre feature set comprises a plurality of timbre features when the target object pronounces a plurality of pinyins; inputting text data and the timbre feature set into a target acoustic model to obtain mel-frequency spectrum data of the target object; and inputting the mel-frequency spectrum data into a target vocoder to obtain target audio corresponding to the text data, wherein the target audio has the timbre feature of the target object. Some embodiments of the present application can synthesize target audio that is more suitable for a target object, and provide rich timbre selection for novel reading, film dubbing and game dubbing.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to an audio synthesis method and device, a storage medium and an electronic device. BACKGROUND

[0002] In recent years, with the rapid development of chip computing power and deep learning algorithms, speech synthesis technology has gradually matured. Current speech synthesis is widely used in real scenarios such as intelligent customer service, telephone sales, voice assistants, and voice navigation.

[0003] Currently, in order to realize the cloning of speech timbre, a multi-timbre speech synthesis model is usually trained. When a customer needs to customize a new timbre, a similar factor can be selected from the timbre library of the multi-timbre speech synthesis model and returned to the customer. Although this method can quickly provide the customer with a customized timbre, due to the limited inventory of the timbre library of the multi-timbre speech synthesis model, it cannot meet the needs of all customers, and thus cannot realize the synthesis of speech timbre.

[0004] Therefore, how to provide a technical solution of an efficient audio synthesis method has become a technical problem to be solved. SUMMARY

[0005] Some embodiments of the present application aim to provide an audio synthesis method, device, storage medium and electronic device. The technical solution of the embodiments of the present application can synthesize target audio close to the timbre characteristics of different target objects, and the audio synthesis effect is good, the synthesis efficiency is high, and the accuracy is high.

[0006] In a first aspect, some embodiments of the present application provide an audio synthesis method, comprising: obtaining a timbre characteristic set corresponding to speech audio of a target object, wherein the timbre characteristic set includes a plurality of timbre characteristics when the target object pronounces a plurality of pinyins; inputting text data and the timbre characteristic set into a target acoustic model to obtain mel-frequency spectrum data of the target object; inputting the mel-frequency spectrum data into a target vocoder to obtain target audio corresponding to the text data, wherein the target audio has the timbre characteristics of the target object.

[0007] Some embodiments of the present application extract the timbre characteristics of the pinyins pronounced by the target object, input the text data and the timbre characteristics of the speech to be synthesized into a target acoustic model, and then obtain target audio close to the timbre characteristics of the target object through a target vocoder. The embodiments of the present application can synthesize target audio close to the timbre characteristics of the target object, and the audio synthesis effect is good, the synthesis efficiency is high, and the accuracy is high.

[0008] In some embodiments, each timbre feature in the set of timbre features comprises a fundamental frequency and at least one formant frequency.

[0009] Some embodiments of the present application can achieve high accuracy of audio synthesis by extracting different timbre features.

[0010] In some embodiments, the set of timbre features corresponding to the speech audio of the target object comprises: segmenting the speech audio to obtain audio segments of each pinyin in the speech audio; extracting the audio segments to obtain timbre features of the plurality of pinyins; statistically analyzing the timbre features of the plurality of pinyins to obtain remaining audio data in a pinyin database other than the audio segments, wherein the pinyin database comprises timbre features of all pinyins under different tones; predicting predicted timbre features corresponding to the target object speaking the remaining audio data; and constructing the set of timbre features by combining the timbre features of the plurality of pinyins and the predicted timbre features.

[0011] Some embodiments of the present application can obtain timbre features of all pinyins spoken by the target object by segmenting, extracting, statistically analyzing, and predicting the speech audio of the target object, thereby providing data support for synthesizing target audio with high accuracy.

[0012] In some embodiments, before the text data and the set of timbre features are input into the target acoustic model, the method further comprises: obtaining a first training data set, wherein the first training data set comprises timbre feature vectors and audio mel-frequency spectrum features corresponding to a plurality of speech contents of a plurality of objects; and training an initial acoustic model using the first training data set to obtain the target acoustic model.

[0013] Some embodiments of the present application can obtain a target acoustic model by using a first training data set and an initial acoustic model, thereby providing model support for obtaining target audio with high accuracy.

[0014] In some embodiments, the structure of the initial acoustic model comprises a phoneme embedding vector module, and the phoneme embedding vector module comprises a plurality of phoneme predictors, wherein the number of the phoneme predictors is the same as the number of the timbre feature vectors.

[0015] Some embodiments of the present application can train a target acoustic model corresponding to the timbre feature vectors by setting a plurality of factor predictors in the structure of the initial acoustic model, thereby achieving a better model effect.

[0016] In some embodiments, the obtaining the first training data set comprises: collecting a plurality of speech contents corresponding to the plurality of subjects; dividing each of the plurality of speech contents in a time sequence to obtain a plurality of Chinese character pronunciation audio segments in each of the speech contents; analyzing each of the Chinese character pronunciation audio segments to obtain a plurality of Chinese character pronunciation timbre features, wherein each Chinese character pronunciation timbre feature in the plurality of Chinese character pronunciation timbre features comprises a fundamental frequency, a first formant frequency, a second formant frequency, and a third formant frequency; removing other pronunciation data in a pronunciation database except for each Chinese character pronunciation audio segment in each of the speech contents; obtaining estimated timbre features of the other pronunciation data except for each Chinese character pronunciation audio segment in each of the speech contents; calculating the plurality of Chinese character pronunciation timbre features and the estimated timbre features to obtain a fundamental frequency feature vector, a first formant frequency vector, a second formant frequency vector, and a third formant frequency vector in a timbre feature vector of each of the plurality of subjects; and inputting text corresponding to the plurality of speech contents into an acoustic model to obtain the audio mel-frequency spectrum features.

[0017] Some embodiments of the present application obtain corresponding feature vectors by dividing, analyzing, and estimating missing pronunciation timbre features of a plurality of speech contents of a plurality of subjects, and then obtain audio mel-frequency spectrum features through an acoustic model. The embodiments of the present application can obtain a first training data set with rich data, thereby providing effective data support for training a model.

[0018] In some embodiments, before the inputting the mel-frequency spectrum data into the target vocoder, the method further comprises: obtaining a second training data set, wherein the second training data set comprises audio mel-frequency spectrum features and audio corresponding to a plurality of speech contents of a plurality of subjects; and training an initial vocoder by using the second training data set to obtain the target vocoder.

[0019] Some embodiments of the present application train an initial vocoder by using a second training data set to obtain a target vocoder, thereby providing model support for obtaining a target audio with high accuracy.

[0020] In a second aspect, some embodiments of the present application provide an audio synthesis device, comprising: a data acquisition module configured to obtain a timbre feature set corresponding to speech audio of a target subject, wherein the timbre feature set comprises a plurality of timbre features when the target subject pronounces a plurality of pronunciations; a model processing module configured to input text data and the timbre feature set into a target acoustic model to obtain mel-frequency spectrum data of the target subject; and an audio synthesis module configured to input the mel-frequency spectrum data into a target vocoder to obtain target audio corresponding to the text data, wherein the target audio has the timbre features of the target subject.

[0021] In a third aspect, some embodiments of the present application provide a computer readable storage medium, having stored thereon a computer program, which, when executed by a processor, enables the method of any of the embodiments of the first aspect.

[0022] In a fourth aspect, some embodiments of the present application provide an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, enables the method of any of the embodiments of the first aspect.

[0023] In a fifth aspect, some embodiments of the present application provide a computer program product, comprising a computer program, wherein the computer program, when executed by a processor, enables the method of any of the embodiments of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of some embodiments of the present application, the following will briefly introduce the drawings needed to be used in some embodiments of the present application. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0025] Figure 1 A system diagram of audio synthesis provided by some embodiments of the present application;

[0026] Figure 2 A method flowchart for obtaining a target acoustic model provided by some embodiments of the present application;

[0027] Figure 3 A structural schematic diagram of an initial acoustic model provided by some embodiments of the present application;

[0028] Figure 4 A structural schematic diagram of an initial acoustic model provided by some embodiments of the present application;

[0029] Figure 5 A method flowchart of audio synthesis provided by some embodiments of the present application;

[0030] Figure 6 A device composition block diagram of audio synthesis provided by some embodiments of the present application;

[0031] Figure 7 An electronic device schematic diagram provided by some embodiments of the present application. DETAILED DESCRIPTION

[0032] The technical solutions in the embodiments of the present application will be described below with reference to the drawings in some embodiments of the present application.

[0033] It should be noted that similar reference numerals and letters refer to similar items in the drawings below, and therefore, once an item is defined in one drawing, it need not be further defined and explained in subsequent drawings. Meanwhile, in the description of the present application, the terms "first", "second", and the like are only used to distinguish description, and cannot be understood as indicating or implying relative importance.

[0034] In related technologies, voice synthesis technology is widely used in real scenarios such as intelligent customer service, telephone sales, voice assistants, and voice navigation. A high-quality voice synthesis system often requires tens of hours of high-quality voice and supporting professional text annotation, but the synthesis process is costly and can only synthesize a fixed timbre. Therefore, how to use a small amount of low-quality data to quickly customize a new timbre voice synthesis model has become an urgent market demand and a research hotspot in the current voice synthesis field. Based on this, voice timbre cloning technology has emerged. Voice timbre cloning technology is also called personalized voice synthesis technology. It usually refers to a technology that uses a small amount of target speaker recording data to train a voice synthesis model to simulate the pronunciation characteristics of the target speaker. A qualified timbre cloning voice synthesis system can provide voice synthesis services with corresponding timbres for users through a small amount of voice data, customize new timbres for voice assistants, voice navigation, and educational companions, and provide rich timbre choices for novel reading, film and television dubbing, and game dubbing.

[0035] Currently, existing voice timbre cloning technologies mainly include the following three categories:

[0036] The first category is to train a multi-timbre voice synthesis model. When a new timbre needs to be customized, a timbre closest to the customized timbre is selected from the timbre library of the existing multi-timbre voice synthesis model and returned to the customer. The advantage of this method is that it requires a low amount of customized timbre recording (usually only three to ten seconds of recording) to give a stable timbre and high accuracy of pronunciation. However, the inventory of the multi-timbre voice synthesis model timbre library is limited and not diverse enough, and it may not be possible to find a timbre similar to the customer's requirements. A timbre library with a diverse enough timbre will inevitably have a relatively high requirement for the cost of the pre-recording data.

[0037] The second type is to fine-tune a multi timbre model or a single timbre model using a small amount of new timbre data (generally more than twenty minutes of recording). The advantage is that it can better fit the target timbre and to some extent imitate the mood and tone of the target timbre. However, on the other hand, the long recording of the new timbre reduces the willingness of the customer to use it, and the short recording of the new timbre easily leads to the phenomenon of unclear pronunciation of part of the synthesized speech, and each customized new timbre needs to occupy server resources for model fine-tuning training, which increases the cost of model deployment.

[0038] The third type is usually composed of a voiceprint model and a speech synthesis model. That is, a valid voiceprint model is trained in advance to extract the voiceprint features of the speech data, and the voiceprint features are combined to train the speech synthesis model. In this way, when the model is applied to the inference in the actual scene, only the voiceprint features of the target speaker need to be extracted using the trained voiceprint model, and input into the also trained speech synthesis model, so that the speech of the target timbre can be synthesized. This method is expected to achieve the effect of approaching the target timbre with a small amount of recording data (usually three to ten seconds of recording). However, the difficulty of this method lies in how to obtain a suitable voiceprint model. The existing neural network-based voiceprint modeling method is usually guided by a relative learning objective function, aiming to obtain voiceprint features that are similar to the same class and distant from the different class. However, such features focus on describing the difference between timbres, and the extracted feature vectors focus more on describing the exclusive identity information rather than the feature information of the speech timbre itself (even if two similar timbres, the model based on relative learning will try to distinguish them). Obviously, the voiceprint features extracted by the above method as input features are insufficient for the speech synthesis model of timbre cloning. Moreover, in actual application, the timbre cloning model based on this method often has problems such as unsimilar synthesized timbre, unclear pronunciation, and poor audio quality.

[0039] From the above related technologies, it can be seen that the existing audio synthesis scheme has the problems of high cost and poor synthesized audio effect.

[0040] Therefore, some embodiments of the present application provide an audio synthesis method. The method extracts timbre features from the speech audio of a target object. Then, the text data and the audio feature set of the synthesized audio are input into a target acoustic model to obtain mel-spectrum data. Finally, the mel-spectrum data is input into a target vocoder to obtain the target audio. Some embodiments of the present application can synthesize target audio similar to the target object by a small amount of target corresponding speech audio, and have good synthesized audio effect, low cost, and wide applicability.

[0041] The overall structure of the system for synthesizing audio provided by some embodiments of the present application will be described below with reference to the accompanying drawings. Figure 1 The overall structure of the system for synthesizing audio provided by some embodiments of the present application will be described below with reference to the accompanying drawings.

[0042] As Figure 1 shown, some embodiments of the present application provide an audio synthesis system, which comprises a terminal 100 and an audio processing server 200. The terminal 100 can store accurate and noise-free voice audio of a target object speaking according to a specified text. The audio processing server 200 can obtain the voice audio and process the voice audio to obtain a timbre feature set of each pinyin when the target object speaks the text in the specified text. Then the audio processing server 200 inputs the text data of the audio to be synthesized and the timbre feature set into a target acoustic model to obtain mel-spectrum data. Finally, the mel-spectrum data is input into a target vocoder to obtain a target audio corresponding to the text data of the audio to be synthesized, which fits the timbre of the target object.

[0043] In some embodiments of the present application, the target acoustic model and the target vocoder trained are pre-deployed in the audio processing server 200. The terminal 100 can be a mobile terminal device or a non-portable computer terminal. The embodiments of the present application are not limited here.

[0044] It should be noted that before performing the audio synthesis method of the present application, the target acoustic model and the target vocoder need to be obtained first.

[0045] The implementation process of obtaining the target acoustic model provided by some embodiments of the present application will be described below in conjunction with the Figure 2 example.

[0046] Please refer to the Figure 2 , Figure 2 flowchart of the method for obtaining the target acoustic model provided by some embodiments of the present application.

[0047] The above process will be described below.

[0048] S210, obtaining a first training data set, wherein the first training data set comprises timbre feature vectors and audio mel-spectrum features corresponding to a plurality of speech contents of a plurality of objects.

[0049] For example, in some embodiments of the present application, before training the initial acoustic model, a first training data set needs to be prepared first.

[0050] In some embodiments of the present application, S210 can include:

[0051] S211, collecting a plurality of speech contents corresponding to the plurality of objects.

[0052] For example, in some embodiments of the present application, a plurality of speech contents of a plurality of speakers (i.e. a plurality of objects) under different Chinese character pronunciation conditions are collected in advance.

[0053] S212, segmenting each of the plurality of voice contents in time sequence to obtain each Chinese character pronunciation audio segment in each of the voice contents.

[0054] For example, in some embodiments of the present application, the time stamp corresponding to the voice content is obtained (i.e., time sequence), and the audio segment corresponding to each Chinese character pronunciation is segmented by the time stamp (i.e., each Chinese character pronunciation audio segment).

[0055] S213, analyzing each Chinese character pronunciation audio segment to obtain the voice characteristics of the plurality of Chinese character pronunciations, wherein the voice characteristics of each Chinese character pronunciation in the voice characteristics of the plurality of Chinese character pronunciations include: fundamental frequency, first formant frequency, second formant frequency, and third formant frequency.

[0056] For example, in some embodiments of the present application, since some Chinese characters have different characters but the same pronunciation, for this case, the fundamental frequency, the first formant frequency, the second formant frequency, and the third formant frequency of the audio segment corresponding to each Chinese character pronunciation are analyzed by the cepstrum method. For the repeated pronunciations (i.e., Chinese characters with the same pronunciation, such as meaning, meaning, and easy), first, the voice characteristics of each character are obtained, and then the voice characteristics of the pronunciation of the Chinese character are obtained by the method of taking the average value (i.e., one voice characteristic corresponding to the pronunciations of the three Chinese characters meaning, meaning, and easy). For example, there are three characters with the same pronunciation, and three fundamental frequencies of the three characters are obtained by analysis, and then the average value of the three fundamental frequencies is obtained to obtain the fundamental frequency of the pronunciation. Assuming that there are n different pronunciations of Chinese pinyin composed of pinyin letters, the voice characteristics of each person can be represented by the fundamental frequency, the first formant frequency, the second formant frequency, and the third formant frequency, as shown in Table 1.

[0057] Table 1

[0058] a1 a2 a3 a4 a5 ... zuo4 zuo5 Pitch frequency [F0(i,0)] [F0(i,1)] [F0(i, 2)] [F0(i, 3)] [F0(i, 4)] ... F0(i,n-1) [F0(i,n)] First formant frequency [F1(i,0)] [F1(i,1)] [F1(i,2)] [F1(i,3)] [F1(i,4)] ... [F1(i,n-1)] [F1(i,n)] Second formant frequency [F2(i,0)] [F2(i,1)] [F2(i,2)] [F2(i,3)] [F2(i,4)] ... F2(i,n-1) F2(i,n) <!-- 5 -->]]> Third formant frequency F3(i,0) F3(i,1) F3(i,2) [F3(i,3)] [F3(i,4)] ... F3(i,n-1) [F3(i,n)]

[0059] In Table 1, "1, 2, 3, 4, 5" represent the tones of Chinese pinyin as 1 tone, 2 tone, 3 tone, 4 tone, and light tone, respectively. In the second to fifth rows, i represents the ith speaker, and n is the total number of Chinese pinyin. Therefore, F0(i, j) can be used to represent the fundamental frequency of the ith speaker when pronouncing the jth Chinese pinyin.

[0060] It should be noted that the fundamental frequency is often used to describe the characteristics of the sound source, such as the thickness, tightness, size of the vocal cords, which represents the brightness or depth of the voice, and is the basis for constructing the speaker's timbre characteristics. The formant frequency reflects the characteristics of the vocal tract, such as the shape characteristics of the oral cavity, nasal cavity, etc. There are differences in vocal cords, oral cavities, and nasal cavities between people, which leads to differences in speech timbre between people. Although everyone's voice can remain stable on a long time scale, the fundamental frequency and formant frequency will fluctuate due to the influence of pronunciation content. For example, the vocal cord shape needs to change to produce different tones, and the oral cavity and nasal cavity need to change to produce different vowels. Therefore, as much as possible, the fundamental frequency and formant frequency of the speaker under different Chinese pronunciations are collected to express the timbre characteristics. In some embodiments of the present application, Montreal Forced Aligner (MFA) can be used to perform forced alignment of the speech data set using Kaldi.

[0061] S214, removing other pinyin data in the pinyin database except for the pinyin audio segment of each Chinese character in each speech content.

[0062] For example, in some embodiments of the present application, insufficient data often leads to difficulty in collecting all different pinyin timbre characteristics of a specific speaker. Since the pinyin database contains n different pronunciations, the frequency characteristics of the missing other pronunciations (as a specific example of other pinyin data) can be obtained by analyzing the collected speech content.

[0063] S215, obtaining estimated timbre characteristics for estimating other pinyin data except for the pinyin audio segment of each Chinese character in each speech content.

[0064] For example, in some embodiments of the present application, the frequency characteristics of the missing other pronunciations of the speaker are estimated by using a matrix factorization algorithm. Taking the fundamental frequency as an example, the fundamental frequencies of different speakers are shown in Table 2.

[0065] Table 2

[0066] a1 a2 a3 a4 a5 ... zuo4 zuo5 Speaker 0 [F0(0,0)] [F0(0,1)] ？ [F0(0, 3)] [F0(0, 4)] ... F0(0, n-1) ？ Speaker 1 F0(1,0) F0(1,1) F0(1,2) F0(1,3) ？ ... F0(1, n-1) F0(1, n) Speaker 2 F0(2,0) ？ F0(2,2) [F0(2,3)] [F0(2,4)] ... ？ F0(2, n) Speaker 3 F0(3,0) [F0(3, 1)] [F0(3, 2)] ？ [F0(3,4)] ... F0(3, n-1) F0(3, n) Speaker 4 ？ F0(4, 1) F0(4,2) F0(4,3) F0(4,4) F0(4, n-1) ？

[0067] In Table 2, the fundamental frequencies of 5 speakers are counted, and "?" represents the missing fundamental frequency of the pinyin. The missing fundamental frequency of the pinyin of each speaker can be estimated by the matrix factorization algorithm to obtain the fundamental frequency of each speaker under different pronunciations. It should be noted that the formant frequency (for example, the first formant frequency, the second formant frequency, or the third formant frequency) of each speaker can also be estimated for the missing pinyin by the above method.

[0068] S216, calculating the pitch frequency feature vector, the first formant frequency vector, the second formant frequency vector and the third formant frequency vector in the timbre feature vector of each object in the plurality of objects based on the timbre features of the plurality of Chinese character pronunciations and the estimated timbre features.

[0069] For example, in some embodiments of the present application, after obtaining the pitch frequency features of each speaker as shown in Table 2, the pitch frequency feature vector of each speaker can be obtained by calculating the feature values of a1~zuo5. Similarly, the first formant frequency vector, the second formant frequency vector and the third formant frequency vector of each speaker can be obtained.

[0070] S217, inputting the text corresponding to the plurality of voice contents into an acoustic model to obtain the audio mel-spectrum features.

[0071] For example, in some embodiments of the present application, inputting the text corresponding to the voice contents of the plurality of speakers into the acoustic model can obtain the audio mel-spectrum features in the first training data set.

[0072] S220, training the initial acoustic model using the first training data set to obtain the target acoustic model.

[0073] In some embodiments of the present application, the structure of the initial acoustic model includes a phoneme embedding vector module, and the phoneme embedding vector module includes a plurality of phoneme predictors, wherein the number of the phoneme predictors is the same as the number of the timbre feature vectors.

[0074] For example, in some embodiments of the present application, the structure of the initial acoustic model is as shown in Figure 3 The structure of the initial acoustic model includes Phoneme (the timbre feature vector of each speaker input), Phoneme Embedding (phoneme embedding module), Positional Encoding (position encoding module), Frequency Adaptor (phoneme embedding vector module), Encoder (encoder), Variance Adaptor (speech space variable encoding), Positional Encoding, Mel-spectrogram Decoder (mel-spectrum decoder for outputting mel-spectrum data graph). Different from the conventional acoustic model, the Frequency Adaptor includes a plurality of phoneme predictors as shown in Figure 4 Figure 4 ​F0Predictor, F1 Predictor, F2 Predictor, F3 Predictor and F Predictor in the F0 Vector, F1 Vector, F2 Vector, F3 Vector are the pitch feature vector, the first formant frequency vector, the second formant frequency vector and the third formant frequency vector (collectively referred to as the timbre feature vector) of each speaker obtained above.

[0075] In some embodiments of the present application, the timbre feature vector of each speaker is taken as an input of an initial acoustic model, and the audio mel-spectrum feature is taken as an output of the initial acoustic model to train the target acoustic model. Figure 3

[0076] In some embodiments of the present application, a process for obtaining a target vocoder is also provided. The method for obtaining the target vocoder comprises: obtaining a second training data set, wherein the second training data set comprises audio mel-spectrum features and audio corresponding to a plurality of speech contents of a plurality of objects; and training an initial vocoder by using the second training data set to obtain the target vocoder.

[0077] For example, in some embodiments of the present application, the vocoder is used to convert the mel-spectrum feature generated by the acoustic model into audio data. For example, the audio mel-spectrum feature corresponding to the speech content of each speaker is taken as an input of an initial vocoder, and the audio is taken as an output of the initial vocoder to train the target vocoder.

[0078] The following will be described in conjunction with the accompanying drawings Figure 5 The implementation process of the audio synthesis performed by the audio processing server 200 provided by some embodiments of the present application is exemplarily described.

[0079] Please refer to the accompanying drawings Figure 5 , Figure 5 A method flowchart of audio synthesis provided by some embodiments of the present application is provided, and the method of audio synthesis comprises: S510, obtaining a timbre feature set corresponding to the speech audio of a target object, wherein the timbre feature set comprises a plurality of timbre features when the target object pronounces a plurality of pinyins. S520, inputting text data and the timbre feature set into a target acoustic model to obtain mel-spectrum data of the target object. S530, inputting the mel-spectrum data into a target vocoder to obtain target audio corresponding to the text data, wherein the target audio has the timbre feature of the target object.

[0080] ​For example, in some embodiments of the present application, first, the correct and noise-free speech audio of the target object according to the given text input is obtained, and the speech audio duration can be between five seconds and ten seconds. Then the timbre feature set is extracted. Second, the text (that is, the text data) that needs to be synthesized is obtained, and the text and the timbre feature set are input into the target acoustic model obtained by the method provided above to obtain the mel-spectrogram data. Finally, the mel-spectrogram data is input into the target vocoder obtained by the method provided above to obtain the target audio. The corresponding literal content of the target audio is the text that needs to be synthesized. The timbre of the target audio is consistent with the timbre of the target object. Figure 2 In the target acoustic model obtained by the method provided above, the mel-spectrogram data is obtained. Finally, the mel-spectrogram data is input into the target vocoder obtained by the method provided above to obtain the target audio. The corresponding literal content of the target audio is the text that needs to be synthesized. The timbre of the target audio is consistent with the timbre of the target object.

[0081] In some embodiments of the present application, each timbre feature in the timbre feature set includes a fundamental frequency and at least one formant frequency.

[0082] For example, in some embodiments of the present application, the fundamental frequency and the formant frequency can be used as the timbre feature parameters. The formant frequency can be set to one or more according to the sound production of the target object. The present application does not make specific limitations here.

[0083] In some embodiments of the present application, S510 can include:

[0084] S511, the speech audio is segmented to obtain each audio segment of each pinyin in the speech audio.

[0085] For example, in some embodiments of the present application, the MFA tool is used to segment the speech audio of the target object according to the time stamp to obtain each audio segment of each pinyin. For example, the speech audio is “I want to go to Beijing to watch the flag. If you want to watch the flag, you must catch an early train, because the flag-raising time is consistent with the sunrise time”. The pronunciation time stamp of each character is segmented, and a total of 34 characters are obtained. 34 audio segments.

[0086] S512, the audio segments are extracted to obtain the timbre features of the plurality of pinyins;

[0087] For example, in some embodiments of the present application, the cepstrum method is used to extract features from each audio segment, and the timbre features of multiple pinyins with different pronunciations are obtained. For multiple pinyins with the same pronunciation (e.g., the same pronunciation of the words in a sentence, such as, yu, yü, and yü), the timbre features of each pinyin need to be extracted and then the average value is taken as the timbre feature of the pinyin. For example, in the above 34 audio segments, the pinyins with the same pronunciation or repeated pinyins are "xiang, kan, sheng, qi, yao, shi, jian, and yi". Taking the word "xiang" as an example, the fundamental frequency and the formant frequency of the two "xiang" audio segments are obtained by extraction, and the average value of the fundamental frequency of the timbre features of the two "xiang" is solved to obtain the fundamental frequency of the pinyin "xiang", and the formant frequency of the pinyin "xiang" can be obtained in the same way.

[0088] S513, statistics of the timbre features of the multiple pinyins are performed, and the remaining audio data of the pinyins other than the multiple pinyins in the pinyin database are obtained, wherein the pinyin database includes the timbre features of all pinyins under different tones.

[0089] For example, in some embodiments of the present application, the fundamental frequency and the formant frequency of the multiple pinyins are statistically obtained to obtain the remaining audio data of other pinyins of the target object which are not collected.

[0090] S514, the predicted timbre features corresponding to the target object pronouncing the remaining audio data are predicted.

[0091] For example, in some embodiments of the present application, the matrix decomposition algorithm is used to predict the timbre features of the missing remaining audio data of the target object, and the predicted timbre features are obtained. The predicted timbre features can complement the timbre features of the pinyins missing in the target object.

[0092] S515, the timbre features of the multiple pinyins and the predicted timbre features constitute the timbre feature set.

[0093] For example, in some embodiments of the present application, the timbre features of the multiple pinyins and the predicted timbre features obtained by the above method can include the timbre features of all pinyins uttered by the target object, and these data constitute the timbre feature set.

[0094] As can be seen from the above some embodiments of the present application, the present application can obtain the target audio close to the timbre features of the target speaker by using a small amount of audio data (about 5 to 10 seconds) of the target speaker (i.e., the target object), and the cost is low on the basis of ensuring efficiency and accuracy.

[0095] Please refer to Figure 6 , Figure 6A constituent block diagram of an apparatus for audio synthesis is shown. It should be understood that the apparatus for audio synthesis corresponds to the method embodiments described above, and can perform each step involved in the method embodiments described above. The specific functions of the apparatus for audio synthesis can be found in the description above, and detailed description is appropriately omitted here to avoid repetition.

[0096] Figure 6 The apparatus for audio synthesis comprises at least one software function module stored in the form of software or firmware in the memory or solidified in the apparatus for audio synthesis, and comprises: a data acquisition module 610 configured to acquire a timbre feature set corresponding to speech audio of a target object, wherein the timbre feature set comprises a plurality of timbre features when the target object pronounces a plurality of pinyins; a model processing module 620 configured to input text data and the timbre feature set into a target acoustic model to obtain mel spectrum data of the target object; and an audio synthesis module 630 configured to input the mel spectrum data into a target vocoder to obtain target audio corresponding to the text data, wherein the target audio has the timbre features of the target object.

[0097] In some embodiments of the present application, each timbre feature in the timbre feature set comprises a fundamental frequency and at least one formant frequency.

[0098] In some embodiments of the present application, the data acquisition module 610 is configured to: segment the speech audio to obtain audio segments of each pinyin in the speech audio; extract the audio segments to obtain timbre features of the plurality of pinyins; statistically analyze the timbre features of the plurality of pinyins to obtain remaining audio data in a pinyin database other than the audio segments, wherein the pinyin database comprises timbre features of all pinyins under different tones; predict predicted timbre features corresponding to the target object pronouncing the remaining audio data; and construct the timbre feature set by combining the timbre features of the plurality of pinyins and the predicted timbre features.

[0099] In some embodiments of the present application, before the model processing module 620, the apparatus for audio synthesis further comprises a first training module (not shown in the figure) configured to: acquire a first training data set, wherein the first training data set comprises timbre feature vectors and audio mel spectrum features corresponding to a plurality of speech contents of a plurality of objects; and train an initial acoustic model by using the first training data set to obtain the target acoustic model.

[0100] In some embodiments of the present application, the structure of the initial acoustic model comprises a phoneme embedding vector module, and the phoneme embedding vector module comprises a plurality of phoneme predictors, wherein the number of the phoneme predictors is the same as the number of the timbre feature vectors.

[0101] In some embodiments of the present application, a first training module (not shown in the figure) is configured to:

[0102] collecting a plurality of voice contents corresponding to the plurality of objects;

[0103] segmenting each of the plurality of voice contents in a time sequence to obtain each Chinese character pronunciation audio segment in each of the plurality of voice contents;

[0104] analyzing each Chinese character pronunciation audio segment to obtain a plurality of Chinese character pronunciation timbre features, wherein each Chinese character pronunciation timbre feature in the plurality of Chinese character pronunciation timbre features comprises a fundamental frequency, a first formant frequency, a second formant frequency, and a third formant frequency;

[0105] excluding other pronunciation data in a pronunciation database from the each Chinese character pronunciation audio segment in each of the plurality of voice contents;

[0106] obtaining estimated timbre features of the other pronunciation data in addition to the each Chinese character pronunciation audio segment in each of the plurality of voice contents;

[0107] calculating the plurality of Chinese character pronunciation timbre features and the estimated timbre features to obtain a fundamental frequency feature vector, a first formant frequency vector, a second formant frequency vector, and a third formant frequency vector in a timbre feature vector of each of the plurality of objects;

[0108] inputting texts corresponding to the plurality of voice contents into an acoustic model to obtain the audio mel-frequency spectrum features.

[0109] In some embodiments of the present application, before the audio synthesis module 630, the audio synthesis device further comprises a second training module (not shown in the figure) configured to obtain a second training data set, wherein the second training data set comprises audio mel-frequency spectrum features and audio corresponding to a plurality of voice contents of a plurality of objects, and train an initial vocoder using the second training data set to obtain the target vocoder.

[0110] Some embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, wherein the program is executable by a processor to implement the operations of the method corresponding to any of the above embodiments.

[0111] Some embodiments of the present application also provide a computer program product, which comprises a computer program, wherein the computer program is executable by a processor to implement the operations of the method corresponding to any of the above embodiments.

[0112] AsFigure 7 As shown, some embodiments of the present application provide an electronic device 700, comprising a memory 710, a processor 720, and a computer program stored in the memory 710 and capable of running on the processor 720, wherein the processor 720 reads the program from the memory 710 through a bus 730 and implements the method of any of the above embodiments when executing the program.

[0113] The processor 720 can process digital signals and can include various computing structures. For example, a complex instruction set computer structure, a reduced instruction set computer structure, or a structure implementing a combination of multiple instruction sets. In some examples, the processor 720 can be a microprocessor.

[0114] The memory 710 can be used to store instructions executed by the processor 720 or data related to the execution of the instructions. These instructions and / or data can include code for implementing some or all of the functions of one or more modules described in the embodiments of the present application. The processor 720 of the embodiments of the present disclosure can be used to execute the instructions in the memory 710 to implement the above-described methods. The memory 710 includes dynamic random access memory, static random access memory, flash memory, optical memory, or other memory well known to those skilled in the art.

[0115] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0116] The above only describes the embodiments of the present application and is not used to limit the protection scope of the present application. For those skilled in the art, the present application can have various changes and modifications. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application. It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0117] It is to be noted that, as used in this specification and the appended claims, the singular forms "a," "an," and "the" include plural referents unless the context clearly dictates otherwise. Thus, for example, reference to "a component" can include a combination of two or more components. Additionally, the terms "comprise," "comprises," and "comprising," or any variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements is not necessarily limited to those elements, but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. Furthermore, unless otherwise indicated herein, the terms "first," "second," "third," etc., are used herein merely as labels, and are not intended to impose ordinal import.

Claims

1. A method for audio synthesis, characterized in that, include: A timbre feature set corresponding to the speech audio of the target object is obtained. The timbre feature set includes multiple timbre features and predicted timbre features when the target object pronounces multiple pinyin. For multiple pinyin with the same pronunciation, the timbre features of each pinyin are extracted and the average value is calculated as the timbre features of the same pinyin. The predicted timbre features are obtained by using a matrix factorization algorithm to predict the timbre features of the remaining audio data missing from the target object in the pinyin database, excluding the multiple pinyin. The text data and the timbre feature set are input into the target acoustic model to obtain the Mel spectrum data of the target object; The Mel spectrum data is input into the target vocoder to obtain the target audio corresponding to the text data, wherein the target audio has the timbre characteristics of the target object.

2. The method as described in claim 1, characterized in that, Each timbre feature in the timbre feature set includes: fundamental frequency and at least one formant frequency.

3. The method as described in claim 1 or 2, characterized in that, The acquisition of the timbre feature set corresponding to the speech audio of the target object includes: The audio is segmented to obtain audio segments of each pinyin syllable in the audio. The timbre features of the multiple pinyin syllables are extracted from each audio segment. The timbre features of the multiple pinyin are statistically analyzed to obtain the remaining audio data in the pinyin database excluding the audio segments. The pinyin database includes the timbre features of all pinyin under different tones. Predict the predicted timbre features corresponding to when the target object speaks the remaining audio data; The timbre features of the multiple pinyin syllables and the predicted timbre features constitute the timbre feature set.

4. The method as described in claim 1 or 2, characterized in that, Before inputting the text data and the timbre feature set into the target acoustic model, the method further includes: Obtain a first training dataset, wherein the first training dataset includes: timbre feature vectors and audio Mel-spectral features corresponding to multiple speech contents of multiple objects; The initial acoustic model is trained using the first training dataset to obtain the target acoustic model.

5. The method as described in claim 4, characterized in that, The structure of the initial acoustic model includes: a phoneme embedding vector module, which includes: multiple phoneme predictors, wherein the number of phoneme predictors is the same as the number of timbre feature vectors.

6. The method as described in claim 4, characterized in that, The process of obtaining the first training dataset includes: Collect multiple voice contents corresponding to the multiple objects; Each of the multiple speech contents is segmented in chronological order to obtain the audio segment of each Chinese character's pinyin in each speech content; The audio segments of each Chinese character's pinyin are analyzed to obtain the timbre features of multiple Chinese character pronunciations. The timbre features of each Chinese character pronunciation include: fundamental frequency, first formant frequency, second formant frequency, and third formant frequency. The pinyin data in the pinyin database excluding the pinyin audio segments of each Chinese character in each speech content; Obtain the estimated timbre features of other pinyin data besides each Chinese character pinyin audio segment in each speech content; The timbre features of the pronunciations of the multiple Chinese characters and the estimated timbre features are calculated to obtain the fundamental frequency feature vector, the first formant frequency vector, the second formant frequency vector and the third formant frequency vector in the timbre feature vector of each of the multiple objects; The text corresponding to the multiple speech contents is input into the acoustic model to obtain the audio Mel-spectral features.

7. The method as described in claim 1 or 2, characterized in that, Before inputting the Mel spectrum data into the target vocoder, the method further includes: Obtain a second training dataset, wherein the second training dataset includes: audio Mel-spectral features and audio corresponding to multiple speech contents of multiple objects; The initial vocoder is trained using the second training dataset to obtain the target vocoder.

8. An audio synthesis apparatus, characterized in that, The apparatus is used to perform the method as described in claim 1, comprising: The data acquisition module is used to acquire the timbre feature set corresponding to the speech audio of the target object, wherein the timbre feature set includes multiple timbre features when the target object pronounces multiple pinyin; The model processing module is used to input text data and the timbre feature set into the target acoustic model to obtain the Mel spectrum data of the target object; An audio synthesis module is used to input the Mel spectrum data into a target vocoder to obtain the target audio corresponding to the text data, wherein the target audio has the timbre characteristics of the target object.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, wherein the computer program, when executed by a processor, performs the method as described in any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored on the memory and running on the processor, wherein the computer program is executed by the processor to perform the method as claimed in any one of claims 1-7.

Images