Whispered speech synthesis method, apparatus, device, and storage medium

Abstract

The application provides a whisper speech synthesis method and device, equipment and a storage medium, wherein the method comprises: training a text front-end model according to a labeled data set to obtain the text front-end model, then predicting the text by the text front-end model to obtain a predicted phoneme sequence of the text, next training an acoustic back-end model according to the predicted phoneme sequence and whisper speech data corresponding to the predicted phoneme sequence to obtain the acoustic back-end model, next inputting a text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, then the acoustic back-end model can predict based on the target phoneme sequence to obtain the whisper speech of the text to be converted. The application introduces attached noise into the model training process, so that the text front-end model and the acoustic back-end model can synthesize whisper speech with attached noise, greatly improving the intelligibility and naturalness of the converted whisper speech.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of speech synthesis, in particular to a whisper speech synthesis method and device, equipment and a storage medium. BACKGROUND

[0002] With the continuous development and landing of artificial intelligence technology in recent years, speech synthesis technology (Text to Speech, TTS) has been widely applied to content broadcasting, human-computer dialogue, audio book broadcasting and other scenarios, among which the TTS technology can convert text into natural speech flow. At present, the effect of TTS technology can be comparable to that of real human pronunciation, but in the exploration of speech synthesis of human language in special sound states, there is still a lack of systematic, complete and reproducible technical solutions.

[0003] In the prior art, a speech synthesis system mainly includes a front-end part and a back-end part. The front-end part analyzes the input text to extract the speech acoustic features required by the back-end module, and the back-end module completes the prediction and synthesis from text to audio according to the speech acoustic features.

[0004] However, unlike traditional speech synthesis, whisper is a low-energy language signal produced by exciting the sound cavity through the half-open sound door with the help of lung airflow without vibration of the vocal cords, which often does not contain fundamental frequency information. Therefore, directly using the existing technology for whisper speech synthesis has poor naturalness and intelligibility. Therefore, how to improve the synthesis effect of whisper speech has become a problem to be solved. SUMMARY

[0005] The purpose of the present application is to provide a whisper speech synthesis method, device, equipment and storage medium to solve the problem of poor whisper speech synthesis effect in the prior art in view of the shortcomings of the prior art.

[0006] To achieve the above purpose, the technical solution adopted by the present application is as follows:

[0007] In a first aspect, the present application provides a whisper speech synthesis method, comprising:

[0008] training a front-end model according to a labeled data set to obtain a text front-end model, wherein the labeled data set includes a plurality of first labeled text sequences, the first labeled text sequence includes labeled attached noise information, and the labeled attached noise information includes a labeled position of attached noise and a labeled type of attached noise;

[0009] inputting a second labeled text sequence into the text front-end model for prediction to obtain a predicted phoneme sequence, wherein the predicted phoneme sequence is used to represent the acoustic features of the second labeled text sequence, and the acoustic features include the position of the attached noise and the type of the attached noise in the second labeled text sequence.

[0010] combine the predicted phoneme sequence with the whispered audio corresponding to the second annotated text sequence to obtain a back-end training sample, and perform model training according to the back-end training sample to obtain an acoustic back-end model;

[0011] input the text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, and input the target phoneme sequence into the acoustic back-end model to obtain whispered speech of the text to be converted.

[0012] Optionally, the inputting the text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted comprises:

[0013] inputting the text to be converted into the text front-end model, converting the text to be converted into an initial phoneme sequence by the text front-end model, and predicting attached noise of the initial phoneme sequence of the text to be converted to obtain an attached noise position and an attached noise type of the text to be converted, wherein the initial phoneme sequence comprises a plurality of phonemes;

[0014] annotating the attached noise position and the attached noise type in the initial phoneme sequence of the text to be converted by the text front-end model to obtain a target phoneme sequence of the text to be converted.

[0015] Optionally, the predicting attached noise of the phoneme sequence of the text to be converted to obtain an attached noise position and an attached noise type of the text to be converted comprises:

[0016] iterating the initial phoneme sequence, sequentially determining a target phoneme in the initial phoneme sequence, and determining a previous phoneme of the target phoneme and a next phoneme of the target phoneme;

[0017] determining a previous attached noise position and a previous attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the previous phoneme;

[0018] determining a middle attached noise position and a middle attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme;

[0019] determining a next attached noise position and a next attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the next phoneme, wherein the phoneme attribute comprises a sound cavity state, a airflow obstruction point position, a phonetic feature difference, and an acoustic feature of the phoneme.

[0020] Optionally, the performing front-end model training according to the annotated data set to obtain a text front-end model comprises:

[0021] inputting the first annotated text sequence into an initial text front-end model to obtain predicted attached noise information of the first annotated text sequence, the predicted attached noise information comprising a predicted position of the attached noise and a predicted type of the attached noise;

[0022] determining a first loss value of the initial text front-end model according to the predicted attached noise information and the annotated attached noise information;

[0023] iteratively optimizing the initial text front-end model according to the first loss value to obtain the text front-end model.

[0024] Optionally, the model training according to the back-end training sample to obtain an acoustic back-end model comprises:

[0025] inputting the back-end training sample into an initial acoustic back-end model to obtain predicted whispered audio;

[0026] determining a second loss value of the initial acoustic back-end model based on the predicted whispered audio and the whispered audio in the back-end training sample;

[0027] iteratively optimizing the initial acoustic back-end model according to the second loss value to obtain the acoustic back-end model.

[0028] Optionally, the inputting the target phoneme sequence into the acoustic back-end model to obtain the whispered speech of the text to be converted comprises:

[0029] extracting acoustic features from the target phoneme sequence;

[0030] generating the whispered speech of the text to be converted based on the acoustic features of the target phoneme sequence.

[0031] Optionally, before the front-end model training according to the annotated data set to obtain the text front-end model, the method further comprises:

[0032] positioning attached noise in the whispered audio corresponding to the initial text sequence to obtain the position of the attached noise in the initial text sequence;

[0033] classifying the attached noise in the whispered audio corresponding to the initial text sequence to obtain the type of the attached noise in the initial text sequence;

[0034] annotating the position of the attached noise and the type of the attached noise in the initial text sequence to obtain the first annotated text sequence.

[0035] In a second aspect, the present application provides a whispered speech synthesis device, the device comprising:

[0036] The front-end training module is configured to perform front-end model training according to a labeled data set to obtain a text front-end model, the labeled data set including a plurality of first labeled text sequences, the first labeled text sequence including labeled attached noise information, the labeled attached noise information including a labeled position of attached noise and a labeled type of attached noise.

[0037] The prediction module is configured to input a second labeled text sequence into the text front-end model for prediction to obtain a predicted phoneme sequence, the predicted phoneme sequence being used to represent acoustic characteristics of the second labeled text sequence, the acoustic characteristics including a position of attached noise and a type of attached noise in the second labeled text sequence.

[0038] The back-end training module is configured to combine the predicted phoneme sequence and a whispered audio corresponding to the second labeled text sequence to obtain a back-end training sample, and perform model training according to the back-end training sample to obtain an acoustic back-end model.

[0039] The conversion module is configured to input a text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, and input the target phoneme sequence into the acoustic back-end model to obtain whispered speech of the text to be converted.

[0040] Optionally, the conversion module is specifically configured to:

[0041] input the text to be converted into the text front-end model, convert the text to be converted into an initial phoneme sequence by the text front-end model, and predict attached noise of the initial phoneme sequence of the text to be converted to obtain a position of attached noise and a type of attached noise of the text to be converted, the initial phoneme sequence including a plurality of phonemes.

[0042] label the position of attached noise and the type of attached noise in the initial phoneme sequence of the text to be converted by the text front-end model to obtain the target phoneme sequence of the text to be converted.

[0043] Optionally, the conversion module is further specifically configured to:

[0044] perform traversal on the initial phoneme sequence to sequentially determine a target phoneme in the initial phoneme sequence, and determine a previous phoneme of the target phoneme and a next phoneme of the target phoneme.

[0045] determine a previous attached noise position and a previous attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the previous phoneme.

[0046] determine a middle attached noise position and a middle attached noise attribute of the target phoneme according to the phoneme attribute of the target phoneme.

[0047] According to the phoneme attribute of the target phoneme and the phoneme attribute of the subsequent phoneme, a post-attachment noise position and a post-attachment noise attribute of the target phoneme are determined, wherein the phoneme attribute comprises a sound cavity state, a airflow obstruction point position, a phonetic feature difference, and an acoustic feature of the phoneme.

[0048] Optionally, the front-end training module is specifically configured to:

[0049] The first annotated text sequence is input into an initial text front-end model to obtain predicted attachment noise information of the first annotated text sequence, the predicted attachment noise information comprising a position of a predicted attachment noise and a type of the predicted attachment noise.

[0050] According to the predicted attachment noise information and the annotated attachment noise information, a first loss value of the initial text front-end model is determined.

[0051] The initial text front-end model is iteratively optimized according to the first loss value to obtain the text front-end model.

[0052] Optionally, the back-end training module is specifically configured to:

[0053] The back-end training sample is input into an initial acoustic back-end model to obtain predicted whisper audio.

[0054] Based on the predicted whisper audio and the whisper audio in the back-end training sample, a second loss value of the initial acoustic back-end model is determined.

[0055] The initial acoustic back-end model is iteratively optimized according to the second loss value to obtain the acoustic back-end model.

[0056] Optionally, the conversion module is further specifically configured to:

[0057] Acoustic features are extracted from the target phoneme sequence.

[0058] Whisper audio of the text to be converted is generated based on the acoustic features of the target phoneme sequence.

[0059] In a third aspect, the present application provides an electronic device, comprising a processor, a storage medium, and a bus, the storage medium storing machine-readable instructions executable by the processor, the processor and the storage medium communicating through the bus when the electronic device is running, and the processor executing the machine-readable instructions to perform the steps of the whisper audio synthesis method described above.

[0060] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium storing a computer program, the computer program being executed by a processor to perform the steps of the whisper audio synthesis method described above.

[0061] The beneficial effects of the present application are: introducing the attached noise as an acoustic feature into the process of training the text front-end model and the acoustic back-end model, so that the text front-end model and the acoustic back-end model both learn the features of the attached noise, and thus when the trained text front-end model and the acoustic back-end model are used for whisper speech synthesis, the whisper speech with attached noise can be synthesized, greatly improving the intelligibility and naturalness of the converted whisper speech. BRIEF DESCRIPTION OF DRAWINGS

[0062] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present application, and therefore should not be regarded as a limitation on the scope, and for those skilled in the art, other related drawings can also be obtained without creative labor on the basis of these drawings.

[0063] Figure 1 An architecture schematic diagram of a whisper speech synthesis system provided by an embodiment of the present application is shown;

[0064] Figure 2 A flowchart of a whisper speech synthesis method provided by an embodiment of the present application is shown;

[0065] Figure 3 A flowchart of obtaining a target phoneme sequence provided by an embodiment of the present application is shown;

[0066] Figure 4 A flowchart of predicting attached noise provided by an embodiment of the present application is shown;

[0067] Figure 5 A flowchart of training a text front-end model provided by an embodiment of the present application is shown;

[0068] Figure 6 A flowchart of training an acoustic back-end model provided by an embodiment of the present application is shown;

[0069] Figure 7 A flowchart of generating whisper speech provided by an embodiment of the present application is shown;

[0070] Figure 8 A flowchart of obtaining a first annotated text sequence provided by an embodiment of the present application is shown;

[0071] Figure 9 A structure schematic diagram of a whisper speech synthesis device provided by an embodiment of the present application is shown;

[0072] Figure 10 A structure schematic diagram of an electronic device provided by an embodiment of the present application is shown. DETAILED DESCRIPTION

[0073] In order to make the objects, technical solutions and advantages of the embodiments of the present application clearer, the following will be combined with the accompanying drawings for clear and complete description of the technical solutions in the embodiments of the present application. It should be understood that the accompanying drawings in the present application are only for the purpose of illustration and description, and are not used to limit the scope of protection of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportion. The flowchart used in the present application shows the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowchart can not be implemented in sequence, and the steps without logical context relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowchart or one or more operations can be removed from the flowchart under the guidance of the content of the present application.

[0074] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0075] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features declared thereafter, but does not exclude the addition of other features.

[0076] In the prior art, when training a speech synthesis system, noise elimination is performed on the speech samples to improve the extraction accuracy of speech acoustic features.

[0077] However, there are great differences between whispering speech and normal volume speech. The former relies on airflow and friction at various places on the airflow path to produce air noise when speaking, while the latter relies on airflow to vibrate the vocal cords to produce vocal cord sound, but both need to rely on the vocal cavity composed of five resonance cavities of throat, pharynx, mouth, lips and nose to form different sounds. The above human vocal organs are often in a wet state in a normal state, and it is inevitable to collect attached noise when sampling whispering, such as saliva sound in the oral cavity, snot sound in the nasal cavity, lip-tooth sound in the lip cavity, liquid vibration sound under the action of airflow in the laryngeal cavity and pharynx, etc.

[0078] In addition, whisper is a low-energy language signal generated by exciting the acoustic cavity with the help of lung airflow through half-opened sound doors without vibrating vocal cords, and often does not contain fundamental frequency information. Due to the lack of fundamental frequency information, the intelligibility of natural whisper is much lower than that of normal natural speech, and the attached noise plays an important role in distinguishing semantics and improving naturalness in whisper.

[0079] However, in the prior art, the existence of the attached noise will greatly interfere with the acoustic feature extraction of the sampling data when synthesizing speech, so in the normal speech synthesis scheme, the attached noise will be avoided and removed in the sampling and data processing link to improve the accuracy of acoustic feature extraction. When the prior art is directly applied to the scene of whisper speech synthesis, this method will undoubtedly greatly reduce the naturalness and intelligibility of the synthesized whisper speech.

[0080] It can be seen that how to improve the synthesis effect of whisper speech has become a problem to be solved.

[0081] Based on the above problems, the present application provides a whisper speech synthesis method, which introduces the attached noise with important semantic role in whisper into model training as a feature to improve the naturalness of whisper speech synthesis.

[0082] As shown in Figure 1 , it is a schematic diagram of a whisper speech synthesis system given by the present application, the method of the present application can be applied in the system, and Figure 1 , the whisper speech synthesis system includes a text front-end model and an acoustic back-end model. After the model training is completed, the text to be converted is input into the text front-end model, and the phoneme sequence of the text to be converted can be obtained. Next, the acoustic back-end model can synthesize the whisper speech of the text to be converted based on the phoneme sequence.

[0083] Next, in combination with Figure 2 , the whisper speech synthesis method of the present application is further described. The execution subject of the method can be an electronic device, as Figure 2 shown, the method comprises the following steps.

[0084] S201: training a front-end model according to a labeled data set, obtaining a text front-end model, the labeled data set including a plurality of first labeled text sequences, the first labeled text sequence including labeled attached noise information, the labeled attached noise information including: labeled position of the attached noise, labeled type of the attached noise.

[0085] In the present application, the noise can be classified and positioned according to the sound production principle rule in each dimension of the sound production principle feature, and the description data of the whisper sound production feature can be introduced in the form of a text sequence to obtain a labeled data set.

[0086] Optionally, the first labeled text sequence can be a text sequence containing the description data of the whisper sound production feature.

[0087] Optionally, the annotated position of the attached noise can represent the position relationship between the attached noise and the phonetic segment in the text. The annotated type of the attached noise can represent the open state of the acoustic cavity, the airflow obstruction point, the relative time length distance, and the like. The annotated position of the attached noise and the annotated type of the attached noise can jointly represent the acoustic characteristics of the attached noise.

[0088] From the acoustic characteristics and the sound production principle, the processing and classification of the attached noise are in the unit of the minimum phonetic segment, i.e., the phoneme. There can be many classification methods for the type of the attached noise. The first classification method can be to divide the attached noise into open type (airflow not obstructed) and closed type (airflow obstructed) according to the open state of the acoustic cavity. The second classification method can be to divide the closed type into lip, tongue tip, tongue blade, tongue surface, and throat according to the contact position between the sound production organs and the airflow obstruction point, totaling six categories. The third classification method can be to divide the attached noise into front, middle, and rear according to the position relationship between the attached noise and the phonetic segment, totaling three categories. The fourth classification method can be to divide the attached noise according to the relative time length distance between the attached noise and the beginning and end points of the phonetic segment. The type of the attached noise can be obtained based on one or more of the above classification methods.

[0089] Taking the Chinese initial d as an example, the tongue tip sound / d / has the sound production principle that the airflow is obstructed at the contact point between the tongue tip and the gum, the tongue tip and the gum are separated under the impact of the airflow, and the airflow in the acoustic cavity vibrates to produce sound. Therefore, the attached noise corresponding to / d / has the following characteristics: the acoustic cavity is closed, the contact between the tongue tip and the gum causes airflow obstruction, the contact occurs before resonance, and the duration is 3-10 milliseconds. At the same time, in terms of acoustic characteristics, the spectrum at the attached noise of the initial d shows a straight bar of 3-10 milliseconds, the energy is concentrated in the high frequency interval of 5k-10kHz, and the level value at the attached noise is slightly smaller than that of the phonetic segment itself.

[0090] S202: input the second annotated text sequence into the text front-end model for prediction to obtain a predicted phoneme sequence, the predicted phoneme sequence being used to represent the acoustic characteristics of the second annotated text sequence, the acoustic characteristics including the position of the attached noise in the second annotated text sequence and the type of the attached noise.

[0091] As a possible implementation, the second annotated text sequence can be the same annotated text sequence as the first annotated text sequence. In the S201 step, the annotated text sequence is used for training, and the text front-end model can learn the annotated attached noise information in the annotated text sequence. In the S202 step, the text front-end model can ignore the annotated attached noise information and only predict the text sequence in the annotated text sequence to obtain the predicted phoneme sequence of the annotated text sequence.

[0092] As another possible implementation, the second annotated text sequence can be a different annotated text sequence from the first annotated text sequence, the difference being that the first annotated text sequence includes annotated attached noise information and the second annotated text sequence does not include the attached noise information, and the text front-end model can predict the second annotated text sequence to obtain a predicted phoneme sequence of the second annotated text sequence, and the predicted phoneme sequence can represent the attached noise information of the second annotated text sequence.

[0093] A basic phoneme refers to the smallest unit of speech divided according to the natural properties of speech, which is analyzed according to the pronunciation action in a syllable, and one action constitutes a basic phoneme. For example, if the text to be processed is Chinese characters, the basic phonemes can refer to the vowels and consonants in the pinyin corresponding to each character. For example, “wei” has three basic phonemes, and “jian” has four basic phonemes.

[0094] Optionally, the predicted phoneme sequence can include the basic phonemes of each text in the text sequence, the phonemes of the attached noise, and the like, and can also include the description information of the sound, such as tone, prosody, and the like. The predicted phoneme sequence can represent the acoustic features of the second annotated text sequence, and the acoustic features can be the mel-spectrogram of the second annotated text sequence. The acoustic features include the position of the attached noise and the type of the attached noise in the second annotated text sequence predicted by the text front-end model.

[0095] Optionally, the text front-end model can first convert the second annotated text sequence into a phoneme sequence, then predict the attached noise for the phoneme sequence to obtain the position of the attached noise and the type of the attached noise of the second annotated text sequence, and label the predicted attached noise at the position corresponding to the phoneme sequence to obtain the final predicted phoneme sequence of the second annotated text sequence.

[0096] S203: Combine the predicted phoneme sequence with the whispered audio corresponding to the second annotated text sequence to obtain a back-end training sample, and perform model training according to the back-end training sample to obtain an acoustic back-end model.

[0097] In this application, the attached noise in the natural speech stream of the whispered audio can be retained, the predicted phoneme sequence of the second annotated text sequence with the attached noise is one-to-one corresponding to the whispered audio and combined to obtain a back-end training sample.

[0098] Optionally, the back-end training sample can include a combination of multiple predicted phoneme sequences and whispered audio, and the predicted phoneme sequence includes the position of the attached noise and the type of the attached noise predicted by the front-end text sequence. Therefore, based on the whispered audio and the predicted phoneme sequence, the model training can enable the acoustic back-end model to learn more acoustic features of the attached noise, so that the final acoustic back-end model can synthesize whispered speech with attached noise, thereby improving the intelligibility and naturalness of the whispered speech.

[0099] S204: Input the text to be converted into the text front-end model to obtain the target phoneme sequence of the text to be converted, and input the target phoneme sequence into the acoustic back-end model to obtain the whispered speech of the text to be converted.

[0100] Optionally, the text to be converted can be the text that needs to be synthesized into whispered speech. After inputting the text to be converted into the text front-end model, the text front-end model can predict the attached noise in the text to be converted, and mark the attached noise as an acoustic feature in the phoneme sequence to obtain the target phoneme sequence of the text to be converted.

[0101] Optionally, after determining the target phoneme sequence, the acoustic back-end model can convert the target phoneme sequence into whispered speech. Among them, the acoustic back-end model can be a vocoder.

[0102] In the embodiment of the present application, first, the text front-end model is trained according to the labeled data set to obtain the text front-end model. Then, the text front-end model predicts the text to obtain the predicted phoneme sequence of the text. Next, the acoustic back-end model is trained according to the predicted phoneme sequence and the whispered speech data corresponding to the predicted phoneme sequence to obtain the acoustic back-end model. Next, input the text to be converted into the text front-end model, and the target phoneme sequence of the text to be converted can be obtained. Then, the acoustic back-end model can predict based on the target phoneme sequence to obtain the whispered speech of the text to be converted. In the present application, the attached noise is introduced as an acoustic feature into the training process of the text front-end model and the acoustic back-end model, which can enable the model to learn the characteristics of the attached noise. Therefore, when using the trained text front-end model and acoustic back-end model for whispered speech synthesis, whispered speech with attached noise can be synthesized, greatly improving the intelligibility and naturalness of the converted whispered speech.

[0103] Next, in combination with Figure 3 The steps of inputting the text to be converted into the text front-end model to obtain the target phoneme sequence of the text to be converted are described as follows. As Figure 3 shown, the above S204 step includes:

[0104] S301: Input the text to be converted into the text front-end model. The text front-end model converts the text to be converted into an initial phoneme sequence, and predicts the attached noise of the initial phoneme sequence of the text to be converted to obtain the position and type of the attached noise of the text to be converted. The initial phoneme sequence includes multiple phonemes.

[0105] Optionally, the initial phoneme sequence can be a sequence composed of the basic phonemes of the text to be converted. Exemplarily, assume the text to be converted is "Today is a nice day", then the initial phoneme sequence can be "j in t i an sh i g e hao t i an qi".

[0106] In the training stage, the text front-end model has learned the features of the attached noise, so in the use stage, the text front-end model can predict the attached noise of each phoneme in the initial phoneme sequence to obtain the position and type of the attached noise of each phoneme in the initial phoneme sequence.

[0107] S302: Labeling the attached noise position and the attached noise type in the initial phoneme sequence of the text to be converted by the text front-end model to obtain the target phoneme sequence of the text to be converted.

[0108] It should be noted that the target phoneme sequence can be a factor sequence containing the attached noise feature, and the text front-end model can label the attached noise position and the attached noise type in the initial phoneme sequence of the text to be converted to obtain the target phoneme sequence of the text to be converted.

[0109] The following is a step-by-step explanation of the above S301 of predicting the attached noise of the initial phoneme sequence of the text to be converted to obtain the attached noise position and the attached noise type of the text to be converted, the initial phoneme sequence including multiple phonemes, as shown in Figure 4 The above S301 step includes:

[0110] S401: Iterating the initial phoneme sequence to determine the target phoneme in the initial phoneme sequence in turn, and determining the previous phoneme of the target phoneme and the next phoneme of the target phoneme.

[0111] Optionally, the electronic device can iterate the initial phoneme sequence according to the arrangement order of each phoneme in the initial phoneme sequence, and determine the target phoneme in the initial phoneme sequence in turn. For the first phoneme in the initial phoneme sequence, only the next phoneme of the phoneme can be determined, and for the last phoneme in the initial phoneme sequence, only the previous phoneme of the phoneme can be determined.

[0112] S402: According to the phoneme attribute of the target phoneme and the phoneme attribute of the previous phoneme, determining the front attached noise position of the target phoneme and the front attached noise attribute.

[0113] Optionally, the phoneme attribute can describe the sound production feature of the phoneme, so the text front-end model can predict the front attached noise position of the target phoneme and the front attached noise attribute according to the phoneme attribute of the target phoneme and the phoneme attribute of the previous phoneme.

[0114] Optionally, the attached noise position of the target phoneme can include three positions of front, middle and back, and the front attached noise position and the front attached noise attribute can jointly represent the acoustic features of the attached noise. Taking the initial consonant / d / of Chinese as an example, the corresponding front attached noise attribute can be: the vocal cavity is closed, the tongue tip contacts the gingival to block the airflow, the contact occurs before resonance, and the duration is 3-10 milliseconds.

[0115] S403: determining the mid-attachment noise position and the mid-attachment noise attribute of the target phoneme according to the phoneme attribute of the target phoneme.

[0116] Optionally, the mid-attachment noise can be an attachment noise occurring in the target phoneme. For example, when a person pronounces a phoneme in a whispering manner, a noise can occur in the middle of the pronunciation. For example, when the person pronounces the / l / in the initial consonants of Chinese, a noise can occur when the tongue vibrates against the palate. The mid-attachment noise position and the mid-attachment noise attribute can be used to describe the acoustic characteristics of the noise.

[0117] S404: determining the post-attachment noise position and the post-attachment noise attribute of the target phoneme according to the phoneme attribute of the target phoneme and the phoneme attribute of the subsequent phoneme, wherein the phoneme attribute includes the acoustic cavity state, the airflow obstruction point position, the phonetic feature difference, and the acoustic characteristics of the phoneme.

[0118] Optionally, the post-attachment noise position and the post-attachment noise attribute can describe the acoustic characteristics of the attachment noise after the phoneme. For example, when the person pronounces the / sh / and / e / in a whispering manner, a noise can occur after the / sh / due to the airflow obstruction of the tongue and the teeth.

[0119] The phoneme attribute in the foregoing S402-S403 can include the acoustic cavity state, the airflow obstruction point position, the phonetic feature difference, and the acoustic characteristics of the phoneme, wherein the acoustic cavity state can be open or closed, the airflow obstruction point position can be the lips, the tongue tip, the tongue blade, the tongue surface, the uvula, and the throat, the phonetic feature difference can represent the acoustic characteristics of the phonetic segment in which the phoneme is located, such as the formant frequency of a vowel, the transition phonetic feature of a stop consonant, and the like, and the acoustic characteristics of the phoneme can describe the frequency and the duration of the phoneme.

[0120] In the embodiments of the present application, by predicting the attachment noise of each phoneme in the initial phoneme sequence, the target phoneme sequence with the attachment noise information can be obtained, so that the finally generated whispering speech can have the attachment noise, and the intelligibility and naturalness of the whispering speech are improved.

[0121] Next, the step of training the front-end model according to the annotated data set to obtain the text front-end model is described, as shown in FIG. 2. Figure 5 The S201 step includes the following steps.

[0122] S501: inputting the first annotated text sequence into the initial text front-end model to obtain the predicted attachment noise information of the first annotated text sequence, the predicted attachment noise information including the position of the predicted attachment noise and the type of the predicted attachment noise.

[0123] Optionally, the initial front-end text model can first convert the first annotated text sequence into an initial phoneme sequence, which can only include the basic phonemes of the first annotated text, and then predict the position and type of the attached noise for each phoneme in the initial phoneme sequence to obtain the predicted attached noise information of the first annotated text sequence.

[0124] The step of predicting the attached noise of the first annotated text sequence by the initial front-end text model can be the same as the step of predicting the attached noise of the text to be converted by the text front-end model, i.e., the aforementioned S301-S302 steps, which will not be repeated here.

[0125] S502: Determine the first loss value of the initial text front-end model according to the predicted attached noise information and the annotated attached noise information.

[0126] The annotated attached noise information includes the pre-annotated attached noise position and type of the first annotated text sequence, so based on the annotated attached noise information and the predicted attached noise information, the first loss value of the initial text front-end model can be determined.

[0127] S503: Iteratively optimize the initial text front-end model according to the first loss value to obtain the text front-end model.

[0128] In this application, the initial text front-end model can be continuously trained, and the model parameters of the text front-end model can be adjusted according to the first loss value until the calculated first loss value is less than a preset threshold, and the final text front-end model is obtained.

[0129] After obtaining the text front-end model, as one possible implementation, the text front-end model can be used to predict the text to obtain the feature sequence of the attached noise, and the acoustic back-end model can be trained based on the feature sequence of the attached noise and the whispered speech data of the feature sequence.

[0130] As another possible implementation, the pre-annotated back-end training sample can also be used to train the acoustic back-end model in this application, and then Figure 6 The step of training the model according to the back-end training sample in the above S203 to obtain the acoustic back-end model will be described as follows, as shown in the following table: Figure 6 The above S203 step includes:

[0131] S601: Input the back-end training sample into the initial acoustic back-end model to obtain the predicted whispered audio.

[0132] Optionally, the back-end training sample can include a phoneme sequence with attached noise and whispered speech data corresponding to the phoneme sequence, wherein the phoneme sequence with attached noise can be a phoneme sequence predicted by the text front-end model, or a phoneme sequence with attached noise pre-labeled by a user.

[0133] S602: Determine a second loss value of the initial acoustic back-end model based on the predicted whispered audio and the whispered audio in the back-end training sample.

[0134] The initial acoustic back-end model can generate predicted whispered audio for the phoneme sequence in the back-end training sample, and based on the predicted whispered audio generated by the initial acoustic back-end model and the whispered audio in the back-end training sample, the initial acoustic back-end model can determine the second loss value.

[0135] The loss function used to determine the second loss value can be the same as the loss function used to determine the first loss value, or other loss functions can be used to calculate the second loss value, which is not limited in the present application.

[0136] S603: Iteratively optimize the initial acoustic back-end model according to the second loss value to obtain the acoustic back-end model.

[0137] It is worth noting that the initial acoustic model can include multiple parts, such as acoustic sub-model, vocoder, etc., and the present application can also model the attached noise as a special phoneme and integrate it into the training of the acoustic back-end parts to obtain an acoustic back-end model capable of synthesizing whispered speech with attached noise information, thereby improving the naturalness and intelligibility of whispered speech.

[0138] Next, combined with Figure 7 , the step of inputting the target phoneme sequence into the acoustic back-end model in S204 above to obtain the whispered speech of the text to be converted will be described. Figure 7 As shown in the above S204 step includes:

[0139] S701: Extract acoustic features from the target phoneme sequence.

[0140] Optionally, the acoustic model can include multiple parts, such as acoustic sub-model, vocoder, etc., and the acoustic features can be extracted from the target phoneme sequence by the acoustic sub-model in the acoustic back-end model.

[0141] Illustratively, the acoustic features can be the mel-spectrogram predicted by the acoustic sub-model according to the target phoneme sequence.

[0142] S702: Generate whispered speech of the text to be converted based on the acoustic features of the target phoneme sequence.

[0143] Optionally, the whisper speech to be converted can be generated based on the acoustic features by a vocoder in the acoustic backend model, and the text-to-whisper speech conversion is completed.

[0144] The following is a step-by-step explanation of obtaining the first annotated text sequence before training the text frontend model as described above. Figure 8 As shown in S201 above, the steps before include:

[0145] S801: Positioning the attached noise in the whisper audio corresponding to the initial text sequence to obtain the position of the attached noise in the initial text sequence.

[0146] The whisper audio corresponding to the initial text sequence can include independent noise and attached noise. The independent noise refers to the noise existing at the pause of the speech flow, which generally does not have the function of semantic transmission and style retention in whisper speech synthesis, so the audio mute segment can be directly processed conventionally, while the attached noise generally exists in the speech flow and is attached to the front, middle and rear positions of each phoneme, which has important value in improving the naturalness and intelligibility of whisper speech synthesis, so it must be retained.

[0147] In this application, by identifying the pause of the speech flow, the independent noise and the attached noise can be distinguished, and the attached noise can be positioned to obtain the position of the attached noise in the initial text sequence.

[0148] S802: Classifying the attached noise in the whisper audio corresponding to the initial text sequence to obtain the type of the attached noise in the initial text sequence.

[0149] Different attached noises will be classified into different categories according to their different sound characteristics. In this application, the attached noise in the whisper audio can be classified according to the sound characteristics of the attached noise to obtain the type of the attached noise in the initial text sequence.

[0150] S803: Labeling the position of the attached noise and the type of the attached noise in the initial text sequence to obtain the first annotated text sequence.

[0151] Optionally, after determining the type of the attached noise, the type of the attached noise can be labeled at the position of the attached noise to obtain the first annotated text sequence. The position of the attached noise and the type of the attached noise can jointly describe the acoustic characteristics of the attached noise.

[0152] Based on the same inventive concept, the application embodiments also provide a whisper speech synthesis device corresponding to the whisper speech synthesis method. Since the principle of the device in the application embodiments solves the problem similar to the above-mentioned whisper speech synthesis method of the application embodiments, the implementation of the device can be referred to the implementation of the method, and the repeated parts will not be described here.

[0153] For reference Figure 9As shown, a whisper speech synthesis device provided by an embodiment of the present application is shown in the figure, and the device comprises a front-end training module 901, a prediction module 902, a back-end training module 903, and a conversion module 904, wherein:

[0154] The front-end training module 901 is configured to perform front-end model training according to a labeled data set to obtain a text front-end model, wherein the labeled data set comprises a plurality of first labeled text sequences, and the first labeled text sequence comprises labeled attached noise information, and the labeled attached noise information comprises a labeled position of attached noise and a labeled type of attached noise.

[0155] The prediction module 902 is configured to input a second labeled text sequence into the text front-end model to perform prediction and obtain a predicted phoneme sequence, wherein the predicted phoneme sequence is used to represent acoustic features of the second labeled text sequence, and the acoustic features comprise a position of attached noise and a type of attached noise in the second labeled text sequence.

[0156] The back-end training module 903 is configured to combine the predicted phoneme sequence and a whisper audio corresponding to the second labeled text sequence to obtain a back-end training sample, and perform model training according to the back-end training sample to obtain an acoustic back-end model.

[0157] The conversion module 904 is configured to input a text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, and input the target phoneme sequence into the acoustic back-end model to obtain a whisper speech of the text to be converted.

[0158] Optionally, the conversion module 904 is specifically configured to:

[0159] input the text to be converted into the text front-end model, convert the text to be converted into an initial phoneme sequence by the text front-end model, and predict attached noise of the initial phoneme sequence of the text to be converted to obtain an attached noise position and an attached noise type of the text to be converted, wherein the initial phoneme sequence comprises a plurality of phonemes.

[0160] label the attached noise position and the attached noise type in the initial phoneme sequence of the text to be converted by the text front-end model to obtain a target phoneme sequence of the text to be converted.

[0161] Optionally, the conversion module 904 is further specifically configured to:

[0162] perform traversal on the initial phoneme sequence to sequentially determine a target phoneme in the initial phoneme sequence and determine a previous phoneme of the target phoneme and a next phoneme of the target phoneme.

[0163] determine a previous attached noise position and a previous attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the previous phoneme.

[0164] determine the pre-attachment noise position and the pre-attachment noise attribute of the target phoneme according to the phoneme attribute of the target phoneme and the phoneme attribute of the previous phoneme, wherein the phoneme attribute comprises a vocal tract state, a airflow obstruction point position, a phonetic difference, and an acoustic feature of the phoneme.

[0165] determine the pre-attachment noise position and the pre-attachment noise attribute of the target phoneme according to the phoneme attribute of the target phoneme and the phoneme attribute of the previous phoneme, wherein the phoneme attribute comprises a vocal tract state, a airflow obstruction point position, a phonetic difference, and an acoustic feature of the phoneme.

[0166] Optionally, the front-end training module 901 is specifically configured to:

[0167] input the first annotated text sequence into the initial text front-end model to obtain predicted attachment noise information of the first annotated text sequence, the predicted attachment noise information comprising a position of a predicted attachment noise and a type of the predicted attachment noise;

[0168] determine a first loss value of the initial text front-end model according to the predicted attachment noise information and the annotated attachment noise information;

[0169] iteratively optimize the initial text front-end model according to the first loss value to obtain the text front-end model.

[0170] Optionally, the back-end training module 903 is specifically configured to:

[0171] input the back-end training sample into the initial acoustic back-end model to obtain predicted whisper audio;

[0172] determine a second loss value of the initial acoustic back-end model based on the predicted whisper audio and the whisper audio in the back-end training sample;

[0173] iteratively optimize the initial acoustic back-end model according to the second loss value to obtain the acoustic back-end model.

[0174] Optionally, the conversion module 904 is further configured to:

[0175] extract an acoustic feature from the target phoneme sequence;

[0176] generate whisper speech of the to-be-converted text based on the acoustic feature of the target phoneme sequence.

[0177] Optionally, the whisper speech synthesis device further comprises a generation module configured to:

[0178] position attachment noise in the initial text sequence to obtain a position of the attachment noise in the initial text sequence;

[0179] classify the attachment noise in the initial text sequence to obtain a type of the attachment noise in the initial text sequence;

[0180] The position of the attached noise and the type of the attached noise are marked in the initial text sequence to obtain a first marked text sequence.

[0181] The description of the processing flow of each module in the device and the interaction flow between the modules can refer to the related description in the above method embodiments, which will not be described in detail here.

[0182] The embodiment of the present application can make the model learn the features of the attached noise by introducing the attached noise as an acoustic feature into the process of training the text front-end model and the acoustic back-end model, so that the whispered speech with the attached noise can be synthesized when the text front-end model and the acoustic back-end model are used for conversion, greatly improving the intelligibility and naturalness of the converted whispered speech.

[0183] The embodiment of the present application also provides an electronic device, as shown in Figure 10 The electronic device structure schematic diagram provided by the embodiment of the present application includes a processor 1001, a memory 1002 and a bus. The memory 1002 stores machine readable instructions (such as the execution instructions of the front-end training module 901, the prediction module 902, the back-end training module 903 and the conversion module 904 in the device in the above method embodiment, etc.) executable by the processor 1001, when the computer device runs, the processor 1001 and the memory 1002 communicate through the bus, and the machine readable instructions are executed by the processor 1001 to perform the processing of the above whispered speech synthesis method. Figure 9

[0184] The embodiment of the present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by the processor to perform the steps of the above whispered speech synthesis method.

[0185] Those skilled in the art can clearly understand that, for the convenience and brevity of the description, the specific working process of the above described system and device can refer to the corresponding process in the method embodiment, which will not be described in detail in the present application. In the several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented by other ways. The device embodiments described above are only schematic, for example, the division of the modules is only a logical function division, and in actual implementation, there can be another division way, and for example, a plurality of modules or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the shown or discussed mutual ones can be indirect coupling or communication connection through some communication interfaces, devices or modules, which can be electrical, mechanical or other forms.

[0186] ​In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. When the functions are realized in the form of software function units and sold or used as an independent product, they can be stored in a computer readable storage medium. Based on this understanding, the technical solutions of the present application or the parts that essentially contribute to the prior art or the parts of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0187] The above is only a specific embodiment of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application.

Claims

1. A whispered speech synthesis method, characterized by, The method comprises the following steps: performing front-end model training according to a labeled data set to obtain a text front-end model, wherein the labeled data set comprises a plurality of first labeled text sequences, and the first labeled text sequence comprises labeled attached noise information, and the labeled attached noise information comprises a labeled position of attached noise and a labeled type of attached noise; inputting a second labeled text sequence into the text front-end model for prediction to obtain a predicted phoneme sequence, wherein the predicted phoneme sequence is used to represent acoustic characteristics of the second labeled text sequence, and the acoustic characteristics comprise a position of attached noise and a type of attached noise in the second labeled text sequence; combining the predicted phoneme sequence with whispered audio corresponding to the second labeled text sequence to obtain a back-end training sample, and performing model training according to the back-end training sample to obtain an acoustic back-end model; inputting a text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, and inputting the target phoneme sequence into the acoustic back-end model to obtain whispered speech of the text to be converted; the step of inputting the text to be converted into the text front-end model to obtain the target phoneme sequence of the text to be converted comprises: inputting the text to be converted into the text front-end model, converting the text to be converted into an initial phoneme sequence by the text front-end model, and predicting attached noise of the initial phoneme sequence of the text to be converted to obtain a position of attached noise and a type of attached noise of the text to be converted, wherein the initial phoneme sequence comprises a plurality of phonemes; annotating the position of attached noise and the type of attached noise in the initial phoneme sequence of the text to be converted by the text front-end model to obtain the target phoneme sequence of the text to be converted; the step of predicting attached noise of the initial phoneme sequence of the text to be converted to obtain the position of attached noise and the type of attached noise of the text to be converted comprises: iterating the initial phoneme sequence to sequentially determine a target phoneme in the initial phoneme sequence, and determine a previous phoneme of the target phoneme and a next phoneme of the target phoneme; determining a previous attached noise position and a previous attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the previous phoneme; determining a middle attached noise position and a middle attached noise attribute of the target phoneme according to the phoneme attribute of the target phoneme; determining a next attached noise position and a next attached noise attribute of the target phoneme according to the phoneme attribute of the target phoneme and a phoneme attribute of the next phoneme, wherein the phoneme attribute comprises a sound cavity state, a airflow obstruction point position, a phonetic feature difference and an acoustic characteristic of the phoneme.

2. The method of claim 1, wherein, the step of performing front-end model training according to a labeled data set to obtain a text front-end model comprises: inputting a first labeled text sequence into an initial text front-end model to obtain predicted attached noise information of the first labeled text sequence, wherein the predicted attached noise information comprises a position of predicted attached noise and a type of predicted attached noise; determining a first loss value of the initial text front-end model according to the predicted attached noise information and the labeled attached noise information; Iterative optimization is performed on the initial text front-end model according to the first loss value, to obtain the text front-end model.

3. The method of claim 1, wherein, The model training according to the back-end training sample comprises: inputting the back-end training sample into an initial acoustic back-end model to obtain predicted whisper audio; determining a second loss value of the initial acoustic back-end model based on the predicted whisper audio and the whisper audio in the back-end training sample; iterative optimization is performed on the initial acoustic back-end model according to the second loss value, to obtain the acoustic back-end model.

4. The method of claim 1, wherein, The inputting of the target phoneme sequence into the acoustic back-end model to obtain the whisper voice of the text to be converted comprises: extracting acoustic features from the target phoneme sequence; generating the whisper voice of the text to be converted based on the acoustic features of the target phoneme sequence.

5. The method according to any one of claims 1 to 4, characterized in that, Before the front-end model training according to the labeled data set to obtain the text front-end model, the method further comprises: locating the attached noise in the whisper audio corresponding to the initial text sequence to obtain the position of the attached noise in the initial text sequence; classifying the attached noise in the whisper audio corresponding to the initial text sequence to obtain the type of the attached noise in the initial text sequence; labeling the position of the attached noise and the type of the attached noise in the initial text sequence to obtain the first labeled text sequence.

6. A whisper speech synthesis apparatus characterized by comprising: comprises: a front-end training module configured to train a front-end model according to a labeled data set to obtain a text front-end model, wherein the labeled data set comprises a plurality of first labeled text sequences, and the first labeled text sequence comprises labeled attached noise information, and the labeled attached noise information comprises a labeled position of attached noise and a labeled type of attached noise; a prediction module configured to input a second labeled text sequence into the text front-end model for prediction to obtain a predicted phoneme sequence, wherein the predicted phoneme sequence is used to represent acoustic features of the second labeled text sequence, and the acoustic features comprise a position of attached noise and a type of attached noise in the second labeled text sequence; a back-end training module configured to combine the predicted phoneme sequence and whisper audio corresponding to the second labeled text sequence to obtain a back-end training sample, and train a model according to the back-end training sample to obtain an acoustic back-end model; a conversion module configured to input a text to be converted into the text front-end model to obtain a target phoneme sequence of the text to be converted, and input the target phoneme sequence into the acoustic back-end model to obtain a whisper voice of the text to be converted; the conversion module is configured to: input the text to be converted into the text front-end model, convert the text to be converted into an initial phoneme sequence by the text front-end model, and predict attached noise of the initial phoneme sequence of the text to be converted to obtain an attached noise position and an attached noise type of the text to be converted, wherein the initial phoneme sequence comprises a plurality of phonemes; label the attached noise position and the attached noise type in the initial phoneme sequence of the text to be converted by the text front-end model to obtain the target phoneme sequence of the text to be converted; the conversion module is configured to: traversing the initial phoneme sequence, sequentially determining a target phoneme in the initial phoneme sequence, and determining a previous phoneme of the target phoneme and a next phoneme of the target phoneme; determining a pre-attached noise position and a pre-attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the previous phoneme; determining a middle-attached noise position and a middle-attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme; determining a post-attached noise position and a post-attached noise attribute of the target phoneme according to a phoneme attribute of the target phoneme and a phoneme attribute of the next phoneme, wherein the phoneme attribute comprises a sound cavity state, a airflow obstruction point position, a phonetic feature difference, and an acoustic feature of the phoneme.

7. An electronic device, comprising: comprising: a processor, a storage medium, and a bus, the storage medium storing program instructions executable by the processor, when the electronic device is running, the processor and the storage medium communicate through the bus, the processor executes the program instructions to perform the steps of the whisper speech synthesis method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to perform the steps of the whisper speech synthesis method according to any one of claims 1 to 5.

Images