Accent marking generation method, speech synthesis method and related device

Abstract

The present invention provides a method for generating accent annotations, a speech synthesis method, and related devices, relating to the field of speech synthesis. The method first obtains the prosodic boundaries of a preset text and the tone and duration of each syllable in the preset text based on audio data corresponding to the preset text; then extracts acoustic features of the audio data; and finally, generates an accent annotation for each syllable based on the prosodic boundaries, acoustic features, and the tone and duration of each syllable. This reduces the time cost of accent annotation and improves its accuracy, thereby ensuring the rhythm and expressiveness of the speech synthesized by the trained speech synthesis model.

Description

Technical Field

[0001] The present invention relates to the technical field of speech synthesis, and in particular to an accent marking generation method, a speech synthesis method and related devices. Background Art

[0002] With the development of artificial intelligence (AI) technology, text-to-speech (TTS) technology has received more and more attention. TTS technology can be used to convert any text information into corresponding speech.

[0003] Currently, speech synthesis is usually achieved by using a speech synthesis model. However, in the process of training the speech synthesis model, the syllables in the training text need to be manually stressed, which is time-consuming and the accuracy of the stress marking is also limited by the operator's linguistic knowledge, affecting the rhythm and expressiveness of the speech synthesized by the trained speech synthesis model. Summary of the Invention

[0004] In order to overcome the shortcomings of the existing technology, the embodiments of the present invention provide a method for generating accent annotations, a method for speech synthesis, and related devices, which can reduce the time cost of accent annotations, improve the accuracy of accent annotations, and ensure the rhythm and expressiveness of the speech synthesized by the trained speech synthesis model.

[0005] The technical solution of the embodiment of the present invention can be implemented as follows:

[0006] In a first aspect, an embodiment of the present invention provides a method for generating accent annotations, the method comprising:

[0007] Based on audio data corresponding to a preset text, obtaining a prosodic boundary of the preset text and a tone and duration of each syllable in the preset text;

[0008] extracting acoustic features of the audio data;

[0009] A stress mark for each syllable is generated according to the prosodic boundary, the acoustic feature, and the tone and duration of each syllable.

[0010] Optionally, the audio data includes a plurality of speech frames, the acoustic features include a fundamental frequency of each of the speech frames, and the step of generating a stress mark for each syllable based on the prosodic boundary, the acoustic features, and the tone and duration of each syllable includes:

[0011] For each syllable, determining at least one target speech frame corresponding to the syllable based on the duration of the syllable and the generation time of each speech frame, and generating a pitch curve of the syllable based on the fundamental frequencies of all the target speech frames, wherein the generation time of each target speech frame is within the duration of the syllable;

[0012] generating a pitch curve of the syllable based on the fundamental frequencies of all the target speech frames, to obtain a pitch curve of each syllable;

[0013] generating an initial stress mark for each syllable according to the pitch curve of each syllable;

[0014] The initial stress marking of each syllable is processed using the prosodic boundary and the tone of each syllable to obtain the stress marking of each syllable.

[0015] Optionally, the step of generating an initial stress mark for each syllable according to the pitch curve of each syllable includes:

[0016] Performing statistical analysis on the pitch curves of all the syllables to obtain the pitch median and pitch standard deviation;

[0017] For each syllable, an initial stress mark of the syllable is determined according to the pitch median, the pitch standard deviation, and the peak value of the pitch curve of the syllable.

[0018] Optionally, the stress marking includes a first marking, a second marking and a third marking, and the stress levels represented by the first marking, the second marking and the third marking decrease in sequence;

[0019] The step of processing the initial stress mark of each syllable by using the prosodic boundary and the tone of each syllable to obtain the stress mark of each syllable comprises:

[0020] For any syllable to be processed among all the syllables, if the initial stress marking of the syllable to be processed is the first marking or the second marking, processing the initial stress marking of the syllable to be processed according to the tone of the syllable to be processed and the tone of each syllable in the preset text that precedes the syllable to be processed to obtain the stress marking of the syllable to be processed;

[0021] If the initial stress marking of the syllable to be processed is the third marking and the tone of the syllable to be processed is Yangping or Shangsheng, processing the initial stress marking of the syllable to be processed according to the valley value of the pitch curve of the syllable to be processed, the duration of the syllable to be processed, and the prosodic boundary to obtain the stress marking of the syllable to be processed;

[0022] Each of the syllables is traversed to obtain the stress mark of each of the syllables.

[0023] Optionally, the accent marking includes a first marking, a second marking, and a third marking, and the accent levels represented by the first marking, the second marking, and the third marking decrease in sequence. The method further includes:

[0024] Among all the syllables, the syllables with the stress marked as the first mark or the second mark are all regarded as target syllables;

[0025] For each target syllable, based on the peak values ​​of the tones and pitch curves of the adjacent syllables before and after the target syllable in the preset text, it is determined whether to modify the stress marking of the adjacent syllable to the stress marking of the target syllable.

[0026] In a second aspect, an embodiment of the present invention provides a speech synthesis method, the method comprising:

[0027] Obtaining a reference text and linguistic features corresponding to the reference text;

[0028] Acquiring reference audio data and acoustic features corresponding to the reference audio;

[0029] Inputting the reference text, the reference audio data, the linguistic features, and the acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with an accent effect corresponding to the reference text;

[0030] The training samples used to train the speech synthesis model are obtained based on the method for generating accent annotations as described in the first aspect above.

[0031] In a third aspect, an embodiment of the present invention provides a device for generating accent annotations, the device comprising:

[0032] A first acquisition module is configured to acquire, based on audio data corresponding to a preset text, a prosodic boundary of the preset text and a tone and duration of each syllable in the preset text;

[0033] An extraction module, configured to extract acoustic features of the audio data;

[0034] A generating module is configured to generate a stress mark for each syllable according to the prosodic boundary, the acoustic feature, and the tone and duration of each syllable.

[0035] In a fourth aspect, an embodiment of the present invention provides a speech synthesis device, comprising:

[0036] A second acquisition module is used to acquire a reference text and linguistic features corresponding to the reference text;

[0037] The second acquisition module is further configured to acquire reference audio data and acoustic features corresponding to the reference audio data;

[0038] a synthesis module, configured to input the reference text, the reference audio data, the linguistic features, and the acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with an accent effect corresponding to the reference text;

[0039] The training samples used to train the speech synthesis model are obtained based on the method for generating accent annotations as described in the first aspect above.

[0040] In a fifth aspect, the present invention provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the method for generating accent markings as described in the first aspect above, and / or the method for speech synthesis as described in the second aspect above.

[0041] In a sixth aspect, the present invention provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the method for generating accent markings as described in the first aspect above, and / or the method for speech synthesis as described in the second aspect above.

[0042] Compared to the prior art, embodiments of the present invention provide a method for generating accent annotations, a speech synthesis method, and related devices. First, based on the audio data corresponding to a preset text, the prosodic boundaries of the preset text and the tone and duration of each syllable in the preset text are obtained; then, the acoustic features of the audio data are extracted; and finally, based on the prosodic boundaries, acoustic features, and the tone and duration of each syllable, an accent annotation for each syllable is generated. Because the embodiments of the present invention utilize prosodic boundaries, acoustic features, and the tone and duration of each syllable to automatically generate an accent annotation for each syllable, the time cost of accent annotation can be reduced, the accuracy of accent annotation can be improved, and the rhythm and expressiveness of the speech synthesized by the trained speech synthesis model can be guaranteed. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other relevant drawings can be obtained based on these drawings without paying any creative work.

[0044] Figure 1 A schematic diagram of a method for generating accent annotations provided by an embodiment of the present invention Figure 1 ;

[0045] Figure 2 A flowchart illustrating an implementation method of step S103 provided in an embodiment of the present invention;

[0046] Figure 3 An example of a pitch curve provided by an embodiment of the present invention Figure 1 ;

[0047] Figure 4 An example of a pitch curve provided by an embodiment of the present invention Figure 2 ;

[0048] Figure 5 An example of a pitch curve provided by an embodiment of the present invention Figure 3 ;

[0049] Figure 6 An example of a pitch curve provided by an embodiment of the present invention Figure 4 ;

[0050] Figure 7 A schematic diagram of a method for generating accent annotations provided by an embodiment of the present invention Figure 2 ;

[0051] Figure 8 A flowchart of a speech synthesis method provided by an embodiment of the present invention;

[0052] Figure 9 A schematic diagram of the structure of a speech synthesis model provided by an embodiment of the present invention;

[0053] Figure 10 A functional unit block diagram of a device for generating accent annotations provided by an embodiment of the present invention;

[0054] Figure 11 A block diagram of functional units of a speech synthesis device provided by an embodiment of the present invention;

[0055] Figure 12 A schematic block diagram of the structure of a computer device provided in an embodiment of the present invention.

[0056] Icons: 100 - accent annotation generation device; 101 - first extraction module; 102 - extraction module; 103 - generation module; 200 - speech synthesis device; 201 - second acquisition module; 202 - synthesis module; 300 - computer device; 310 - memory; 320 - processor. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions of the embodiments of the present invention will be clearly and completely described below in conjunction with the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Generally, the components of the embodiments of the present invention described and shown in the drawings herein can be arranged and designed in various different configurations.

[0058] Therefore, the following detailed description of the embodiments of the present invention provided in the accompanying drawings is not intended to limit the scope of the invention as claimed, but rather merely represents selected embodiments of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort shall fall within the scope of protection of the present invention.

[0059] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0060] In addition, the terms "first", "second", etc., if used, are merely used to distinguish and describe, and should not be understood as indicating or implying relative importance.

[0061] It should be noted that, in the absence of conflict, the features in the embodiments of the present invention may be combined with each other.

[0062] In order to reduce the time cost of accent annotation, improve the accuracy of accent annotation, and ensure the rhythm and expressiveness of the speech synthesized by the trained sound synthesis model, an embodiment of the present invention provides an accent annotation generation method, which will be described in detail below.

[0063] Please refer to Figure 1 The method for generating accent markings includes steps S101 to S103.

[0064] S101 , based on audio data corresponding to a preset text, obtaining a prosodic boundary of the preset text and the tone and duration of each syllable in the preset text.

[0065] The preset text is the audio recording, and a syllable is the phonetic notation of each character in the preset text. Linguistic features of the audio data corresponding to the preset text can be extracted to obtain the prosodic boundaries of the preset text and the tones of each syllable in the preset text. This embodiment of the present invention does not specifically limit the method for extracting linguistic features.

[0066] The prosodic boundaries are used to indicate where pauses should be made in the preset text. For example, the prosodic boundaries are divided into four pause levels: #1, #2, #3 and #4, with the pause levels increasing in sequence.

[0067] Tone refers to the rise and fall changes of sound. In Mandarin, there are five tones: high level tone (T1), rising tone (T2), falling-rising tone (T3), falling tone (T4), and light tone (T5). In the embodiments of the present invention, T1, T2, T3, T4, and T5 can be used to represent the above tones respectively. It can be understood that for a syllable, such as ni, when it is in different tones, it can be represented as T1[nī], T2[ní], T3[nǐ], T4[nì], and T5[ni].

[0068] For example, the preset text is "你好呀朋友" (Hello, friend), and the linguistic features of its corresponding audio data are ni2hao3#1ya1#2peng2you3#3. From the linguistic features, it can be determined that the tones of the syllables ni, hao, ya, peng, and you are T2, T3, T1, T2, and T3 respectively. The prosodic boundaries of the preset text are #1 between the syllables hao and ya, #2 between the syllables ya and peng, and #3 after the syllable you.

[0069] A syllable is composed of phonemes. For example, the syllable ni is composed of the phonemes n and i. It can be understood that the duration of a syllable is determined by the duration information of the phonemes that make it up, and the duration information of the phonemes can be obtained by using a phoneme alignment tool (MFA).

[0070] S102, extract the acoustic features of the audio data.

[0071] Among them, the audio data includes multiple speech frames, and the acoustic features include the fundamental frequency (f0) of each speech frame. The fundamental frequency can characterize the pitch and can be extracted through the fundamental frequency detection algorithm PDA.

[0072] S103, generate the stress annotation for each syllable according to the prosodic boundary, acoustic features, and the tone and duration of each syllable. <000,0187>Among them, the stress annotation represents whether a syllable is stressed and the stress level. Exemplarily, the stress annotation of a syllable can be obtained by marking 2, 1, or 0 for the syllable. When a syllable is marked as 2, it means that the syllable needs to be strongly stressed; when a syllable is marked as 1, it means that the syllable is regularly stressed; when a syllable is marked as 0, it means that the syllable is unstressed.

[0074] It should be noted that the method in the above embodiments mainly performs stress annotation on each syllable corresponding to the Chinese recording content of the audio data. For the English and mixed Chinese-English recording content, the non-Chinese part is directly set to zero stress, that is, marked as 0, indicating that this part is unstressed.

[0075] Step S103 will be described in detail below.

[0076] Please refer to Figure 2 , step S103 includes sub-steps S103-1 to S103-3.

[0077] S103-1, for each syllable, determine at least one target speech frame corresponding to the syllable according to the duration of the syllable and the generation time of each speech frame, and generate a pitch curve of the syllable based on the fundamental frequencies of all target speech frames.

[0078] The generation time of each target speech frame is within the duration of a syllable.

[0079] Because the fundamental frequency ranges vary between speakers, for example, the fundamental frequency range for male speakers is generally 55-300 Hz, the fundamental frequency range for female speakers is generally 75-450 Hz, and the fundamental frequency range for children is generally 100-700 Hz, the fundamental frequency of each speech frame can be adjusted based on the fundamental frequency range of the speaker in the audio data.

[0080] For example, assuming that the speaker's fundamental frequency ranges from 55 to 300 Hz, then for speech frames with a fundamental frequency greater than 300 Hz, the fundamental frequency of such speech frames is adjusted to 300 Hz. Similarly, for speech frames with a fundamental frequency lower than 55 Hz, the fundamental frequency of such speech frames is adjusted to 55 Hz.

[0081] Generally, human perception of pitch differences is not linear but approximately logarithmic. Therefore, the following formula can be used to normalize the unit of the fundamental frequency of each speech frame from Hertz to semitone.

[0082]

[0083] In the formula, S is the fundamental frequency after processing, in semitones; f0 is the fundamental frequency before processing, in Hertz; f0 ref It is the lower limit of the speaker's fundamental frequency, in Hertz.

[0084] In the embodiment of the present invention, the fundamental frequencies of all target speech frames corresponding to any syllable are processed as described above, and the fundamental frequencies of all target speech frames corresponding to the syllable are fitted as follows: Figure 3 In the pitch curve shown, the horizontal axis of the pitch curve is the duration of the syllable, and the vertical axis of the pitch curve is the fundamental frequency.

[0085] S103-2: Generate an initial stress mark for each syllable based on the pitch curve of each syllable.

[0086] Optionally, step S103-2 may include sub-steps S103-2-1 to S103-2-2.

[0087] S103-2-1, perform statistical analysis on the pitch curves of all syllables to obtain the pitch median and pitch standard deviation.

[0088] The median of pitch refers to the number in the middle position or the average of the two middle numbers after the fundamental frequencies of each time coordinate on the pitch curve of all syllables are arranged from small to large.

[0089] The pitch standard deviation is the square root of the variance of the fundamental frequency at all time coordinates on the pitch curve of all syllables.

[0090] S103-2-2, for each syllable, determine the initial stress marking of the syllable according to the pitch median, the pitch standard value and the peak value of the pitch curve of the syllable.

[0091] The stress marking includes a first marking, a second marking and a third marking, and the stress levels represented by the first marking, the second marking and the third marking decrease in sequence.

[0092] Optionally, the implementation process of S103-2-2 may be as follows:

[0093] For any syllable, the peak value of the pitch curve of the syllable is compared with the peak values ​​of the pitch curves of two adjacent syllables located before and after the syllable in the preset text.

[0094] At this time, there are two possible situations in the comparison results, namely "the peak value of the pitch curve of the syllable is greater than the peak value of the pitch curve of the two adjacent syllables at the same time" and "the peak value of the pitch curve of the syllable is not greater than the peak value of the pitch curve of the two adjacent syllables at the same time".

[0095] In the former case, it is necessary to compare the peak value of the pitch curve of the syllable with the first reference value and the second reference value, and determine the initial stress marking of the syllable based on the comparison result.

[0096] The first reference value and the second reference value are calculated by the pitch median and the pitch standard value according to the following formula.

[0097] A=Med+1.5δ

[0098] B=Med+0.5δ

[0099] In the formula, A is the first reference value, B is the second reference value, Med is the median of the pitch, and δ is the standard deviation of the pitch.

[0100] If the comparison result shows that the peak value of the pitch curve of the syllable is greater than the first reference value, the stress mark of the syllable is set to the first mark.

[0101] If the comparison result is that the peak value of the pitch curve of the syllable is greater than the second reference value and not greater than the first reference value, the stress mark of the syllable is set to the second mark.

[0102] If the comparison result shows that the peak value of the pitch curve of the syllable is not greater than the second reference value, the stress mark of the syllable is set to the third mark.

[0103] For the latter case, that is, the peak value of the pitch curve of the syllable is not greater than the peak values ​​of the pitch curves of the two adjacent syllables, the stress mark of the syllable needs to be set to the third mark.

[0104] For example, the stress mark of a syllable can be set to the first mark, the second mark, or the third mark by marking the syllable as 2, 1, or 0. It can be understood that when a syllable is marked as 2, that is, the stress mark of the syllable is the first mark, indicating that the syllable needs to be strongly stressed; when a syllable is marked as 1, that is, the stress mark of the syllable is the second mark, indicating that the syllable is regularly stressed; when a syllable is marked as 0, that is, the stress mark of the syllable is the third mark, indicating that the syllable is unstressed.

[0105] S103-3, using the prosodic boundary and the tone of each syllable, the initial stress marking of each syllable is processed to obtain the stress marking of each syllable.

[0106] Optionally, step S103-3 may include sub-steps S103-3-1 to S103-3-2.

[0107] S103-3-1, for any syllable to be processed among all syllables, if the initial stress mark of the syllable to be processed is the first mark or the second mark, then the initial stress mark of the syllable to be processed is processed according to the tone of the syllable to be processed and the tone of each syllable before the syllable to be processed in the preset text to obtain the stress mark of the syllable to be processed.

[0108] In an embodiment of the present invention, the implementation process of "processing the initial stress marking of the syllable to be processed according to the tone of the syllable to be processed and the tone of each syllable preceding the syllable to be processed in the preset text" may be as follows:

[0109] Case 1: The tone of the syllable to be processed is rising tone (T3) or light tone (T5).

[0110] At this time, the stress mark of the syllable in the preset text that is located before the syllable to be processed, closest to the syllable to be processed, and whose tone is not T5 needs to be set as the initial stress mark of the syllable to be processed, and the stress mark of the syllable to be processed is set as the third mark.

[0111] Case 2: The tone of the syllable to be processed is Yangping (T2).

[0112] At this point, it is necessary to determine which of the following conditions the syllable to be processed meets:

[0113] (1) The syllable to be processed is not the first syllable in the preset text;

[0114] (2) The syllable to be processed is the first syllable in the preset text;

[0115] (3) The fundamental frequency of each time coordinate in the second half of the pitch curve of the syllable to be processed is less than (0.5*pitch standard deviation).

[0116] When the syllable to be processed meets conditions (1) and (3), the stress mark of the syllable in the preset text that is located before the syllable to be processed, is closest to the syllable to be processed, and has a tone other than T5 is set as the initial stress mark of the syllable to be processed, and the stress mark of the syllable to be processed is set as the third mark;

[0117] When the syllable to be processed meets condition (2), the stress mark of the syllable to be processed is directly set to the third mark.

[0118] It can be understood that, in other cases, the final stress marking of the syllable to be processed is its initial stress marking.

[0119] Case 3: The tone of the syllable to be processed is falling tone (T4).

[0120] At this time, you need to obtain Figure 4 The peak value, peak time coordinate, valley value and valley time coordinate of the pitch curve of the syllable to be processed are shown, and the pitch change rate is calculated using the following formula.

[0121]

[0122] In the formula, slope is the pitch change rate, max is the peak value, min is the valley value, t1 is the peak time coordinate, and t2 is the valley time coordinate.

[0123] It should be noted that if the pitch curve does not have obvious peaks and valleys, the maximum and minimum values ​​in the pitch curve are regarded as peaks and valleys.

[0124] If the pitch change rate is less than 10 and the tone of the adjacent syllable before the syllable to be processed in the preset text is T2 or T3, the stress mark of the adjacent syllable is set to the initial stress mark of the syllable to be processed, and the stress mark of the syllable to be processed is set to the third mark.

[0125] If the pitch change rate is less than 10, and the tone of the adjacent syllable preceding the syllable to be processed in the preset text is not T2 or T3, the stress mark of the syllable to be processed is set to the third mark.

[0126] If the pitch change rate is greater than 10 and the pitch curve peak of the syllable to be processed is less than (pitch median + 0.75*pitch standard deviation), the stress mark of the syllable to be processed is set to the third mark.

[0127] It can be understood that, in other cases, the final stress marking of the syllable to be processed is its initial stress marking.

[0128] Case 4: the tone of the syllable to be processed is yinping (T1).

[0129] At this time, you need to obtain Figure 5 The peak value, peak time coordinate, valley value and valley time coordinate of the pitch curve of the syllable to be processed are shown, and the pitch change rate is calculated using the following formula.

[0130]

[0131] In the formula, slope is the pitch change rate, max is the peak value, min is the valley value, t1 is the peak time coordinate, and t2 is the valley time coordinate.

[0132] It should be noted that if the pitch curve does not have obvious peaks and valleys, the maximum and minimum values ​​in the pitch curve are regarded as peaks and valleys.

[0133] If the pitch change rate is less than 15 and the pitch curve peak of the syllable to be processed is less than (pitch median + 0.75*pitch standard deviation), the stress mark of the syllable to be processed is set to the third mark.

[0134] It can be understood that, in other cases, the final stress marking of the syllable to be processed is its initial stress marking.

[0135] S103-3-2, for any syllable to be processed among all syllables, if the initial stress mark of the syllable to be processed is the third mark and the tone of the syllable to be processed is Yangping (T2) or Shangsheng (T3), then according to the valley value of the pitch curve of the syllable to be processed, the duration of the syllable to be processed and the rhythmic boundary, the initial stress mark of the syllable to be processed is processed to obtain the stress mark of the syllable to be processed.

[0136] In an embodiment of the present invention, the implementation process of “processing the initial stress marking of the syllable to be processed according to the valley value of the pitch curve of the syllable to be processed, the duration of the syllable to be processed, and the prosodic boundary of the syllable to be processed” may be as follows:

[0137] Case 1: The tone of the syllable to be processed is rising tone (T3).

[0138] At this point, it is necessary to determine which of the following conditions the syllable to be processed meets:

[0139] (1) The valley value of the pitch curve of the syllable to be processed is smaller than the valley values ​​of the pitch curves of the two adjacent syllables before and after the syllable to be processed in the preset text;

[0140] (2) the valley value of the pitch curve of the syllable to be processed is the minimum value among the valley values ​​of the pitch curves of all syllables in the preset text;

[0141] (3) The valley value of the pitch curve of the syllable to be processed is the minimum value among the valley values ​​of the pitch curves of all syllables in the largest phrase in which it is located;

[0142] (4) The duration of the syllable to be processed is greater than the duration of the two adjacent syllables;

[0143] (5) The duration of the syllable to be processed is greater than the average duration of all syllables in the preset text.

[0144] When the syllable to be processed meets conditions (1), (2), (4) and (5), the stress mark of the syllable to be processed is modified to the first mark.

[0145] When the syllable to be processed meets conditions (1), (3), (4) and (5), the stress mark of the syllable to be processed is modified to the second mark.

[0146] It should be noted that the maximum phrase in which the syllable to be processed is located is determined by the two prosodic boundaries #3 located before and after the syllable to be processed and closest to the syllable to be processed in the preset text.

[0147] It can be understood that in other cases, the accent mark remains the third mark.

[0148] Case 2: The tone of the syllable to be processed is Yangping (T2).

[0149] At this time, you need to obtain Figure 6 The peak value, peak time coordinate, valley value and valley time coordinate of the pitch curve of the syllable to be processed are shown, and the pitch change rate is calculated using the following formula.

[0150]

[0151] In the formula, slope is the pitch change rate, max is the peak value, min is the valley value, t1 is the valley value time coordinate, and t2 is the peak value time coordinate.

[0152] It should be noted that if the pitch curve does not have obvious peaks and valleys, the maximum and minimum values ​​in the pitch curve are regarded as peaks and valleys.

[0153] Determine which of the following conditions the syllable to be processed meets:

[0154] (1) The pitch change rate is greater than 30;

[0155] (2) the value of t2-t1 is greater than the duration of the syllable to be processed;

[0156] (3) The duration of the syllable to be processed is greater than (1.75*the average duration of all syllables in the preset text);

[0157] (4) The duration of the syllable to be processed is greater than (1.2*the average duration of all syllables in the preset text).

[0158] When the syllable to be processed meets conditions (1), (2) and (3), the stress mark of the syllable to be processed is modified to the first mark.

[0159] When the syllable to be processed meets conditions (1), (2) and (4), the stress mark of the syllable to be processed is modified to the second mark.

[0160] It can be understood that, in other cases, the stress mark of the syllable to be processed remains the third mark.

[0161] Traverse each syllable and execute steps S103-3-1 to S103-3-2 for each syllable to obtain the stress mark of each syllable.

[0162] To further improve the accuracy of the accent marking for each syllable in the preset text, please refer to Figure 7 After step S103, the method for generating accent markings further includes steps S104 to S105.

[0163] S104: syllables with the first or second stress mark in all syllables are taken as target syllables.

[0164] It can be understood that step S105 is not performed for the syllable whose stress mark is the third mark.

[0165] S105 , for each target syllable, judging whether to modify the stress mark of the adjacent syllable to the stress mark of the target syllable based on the tones and peaks of the pitch curves of the adjacent syllables before and after the target syllable in the preset text.

[0166] Optionally, the implementation process of “determining whether to modify the stress marking of the adjacent syllable to the stress marking of the target syllable based on the peak values ​​of the tones and pitch curves of the adjacent syllables before and after the target syllable in the preset text” may be as follows:

[0167] Determine whether the target syllable meets all of the following conditions:

[0168] (1) The tones of the two adjacent syllables before and after the target syllable are both T1 or T4;

[0169] (2) The difference between the peak value of the pitch curve of the target syllable and the peak values ​​of the pitch curves of the two adjacent syllables is less than (0.45*pitch standard deviation);

[0170] (3) The peak value of the pitch curve of the target syllable is greater than (median pitch + 0.45*standard deviation of pitch).

[0171] When the target syllable meets all the above conditions, the stress markings of the two adjacent syllables before and after are modified to the stress markings of the target syllable.

[0172] It is understandable that when the target syllable does not meet all of the above conditions at the same time, there is no need to modify the stress markings of the two adjacent syllables to the stress marking of the target syllable.

[0173] The present invention also provides a method for speech synthesis. Figure 8 The speech synthesis method includes steps S201 to S202.

[0174] S201: Obtain a reference text and linguistic features corresponding to the reference text.

[0175] Among them, the reference text can be a spoken text converted from the voice of any person with a certain identity who wants to change the tone. For example, if A says "I love my motherland" in Mandarin, the reference text can be "I love my motherland" expressed in text form.

[0176] The linguistic features corresponding to the reference text include a tone sequence consisting of the tone of each syllable of the reference text and a prosodic sequence consisting of the prosodic boundaries of the reference text.

[0177] S202: Obtain reference audio data and acoustic features corresponding to the reference audio data.

[0178] The reference audio data may be audio data corresponding to the timbre to be converted, for example, it may be audio data of recording B's voice or audio data of recording C's voice.

[0179] The acoustic features corresponding to the reference audio data may be fundamental frequency features at the frame level.

[0180] S203: Input the reference text, reference audio data, linguistic features, and acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with an accent effect corresponding to the reference text.

[0181] The training samples used to train the speech synthesis model are obtained based on the aforementioned accent annotation generation method.

[0182] like Figure 9As shown, the speech synthesis model includes a front-end model and a back-end model. The training samples obtained based on the aforementioned stress annotation generation method are mainly used to train the front-end model in the speech synthesis model.

[0183] The overall framework of the front-end model is based on the ALBERT model, which incorporates several prior information related to stress, including word pairs found based on the lexicon, prosodic sequences, tone sequences, and discourse-level context information.

[0184] In the word pair information {char, {word1, word2, …, word m}}, word i represents the word vectors of all words containing char in the sentence. For example, in the sentence "Chinese people", for "中" - {中国, 中国人, 中国人}, and for "国" - {中国, 国人, <pad>The word pair information undergoes nonlinear transformation and is introduced into the transformer layer of the ALBERT model.

[0185] Tone embedding: Tone E in each syllable Tone =e T (Tone t ), where embedding e T is a trainable randomly initialized embedding lookup table, Tone t Indicates the tone of the current word at time t. The embedding size is 5 because there are five types of tones: {T1, T2, T3, T4, T5}.

[0186] Prosodic embedding: the prosodic boundary E of each syllable Prosody =e P (Prosody t ), where embedding e P is a trainable randomly initialized embedding lookup table, Prosody t It represents the prosodic boundary of the current syllable at time t. The embedding size is 3 because there are three types of prosodic boundaries: {PW, PPH, IPH}.

[0187] Chapter information embedding: In a similar structural setting of the dialogue context encoder, the sentence window can be set to 2, and the bert embedding of the first two sentences of the current sentence to the current sentence is intercepted as the input of the encoder to obtain the sentence containing {sentence t-2 ,sentence t-1 ,sentence t } output of contextual semantic information.

[0188] Based on word pair information, tone embedding, prosody embedding, and chapter information embedding as features, a dense layer is randomly initialized from the pre-trained Albert-chinese-base model for fine-tuning (fine-tuning, adjustment), and a CRF layer is used for accent labeling.

[0189] During training, the front-end model is tasked with minimizing the sentence-level negative log-likelihood loss.

[0190] In the decoding stage, the input reference text T = {T1, T2, T3, ..., T N } to the front-end model, and obtain a sequence ST consisting of the accent marks of each syllable of the text = {ST1, ST2, ST3, ..., ST N }, where the sequence ST with the best probability is obtained by the Viterbi algorithm.

[0191] The overall framework of the backend model is based on the improved DurIAN model. The reference audio data is passed through an IPA-level accent extractor to obtain the location and degree of accents. For example, in the WAV file: "The weather is really nice today—>^1Today^1day^1day^0the weather^0is^0really^1not^0good–>," the syllable-level accent annotation sequence ST output by the front-end model is aligned to the phoneme level to ultimately obtain IPA-level accent embeddings.

[0192] The 128-dimensional accent embedding is concatenated with the f0 embedding, energy embedding, and the output of the text encoder layer. The information is aligned at the phoneme level according to the phoneme duration information, and then used as the input of the DurIAN decoder to finally obtain audio with accent effect.

[0193] Compared with the prior art, the embodiments of the present invention have the following effects:

[0194] (1) Using prosodic boundaries, acoustic features, and the tone and duration of each syllable, the stress annotation of each syllable is automatically generated, which can reduce the time cost of stress annotation and improve the accuracy of stress annotation;

[0195] (2) The front-end model in the speech synthesis model is trained using the training samples obtained by the accent annotation generation method provided by the embodiment of the present invention to ensure the rhythm and expressiveness of the speech synthesized by the speech synthesis model.

[0196] In order to execute the corresponding steps in the above method embodiment and various possible implementations, an implementation of an accent annotation generating device 100 and an implementation of a speech synthesis device 200 are respectively provided below.

[0197] Please refer to Figure 10 The accent annotation generating device 100 includes a first acquiring module 101 , an extracting module 102 and a generating module 103 .

[0198] The first acquisition module 101 is configured to acquire the prosodic boundaries of the preset text and the tone and duration of each syllable in the preset text based on audio data corresponding to the preset text.

[0199] The extraction module 102 is used to extract acoustic features of audio data.

[0200] The generating module 103 is used to generate the stress mark of each syllable according to the prosodic boundary, acoustic features, and the tone and duration of each syllable.

[0201] Optionally, the audio data includes multiple speech frames, and the acoustic features include the fundamental frequency of each speech frame. The generation module 103 is specifically used to determine at least one target speech frame corresponding to each syllable according to the duration of the syllable and the generation time of each speech frame, and generate a pitch curve of the syllable based on the fundamental frequencies of all target speech frames, wherein the generation time of each target speech frame is within the duration of the syllable; generate an initial stress mark for each syllable according to the pitch curve of each syllable; and process the initial stress mark for each syllable using the rhythmic boundary and the tone of each syllable to obtain the stress mark for each syllable.

[0202] Optionally, when the generation module 103 is used to generate the initial stress mark of each syllable based on the pitch curve of each syllable, it is specifically used to perform statistical analysis on the pitch curves of all syllables to obtain the median of pitch and the standard deviation of pitch; for each syllable, the initial stress mark of the syllable is determined based on the median of pitch, the standard value of pitch and the peak value of the pitch curve of the syllable.

[0203] Optionally, the stress marking includes a first marking, a second marking and a third marking, and the stress levels represented by the first marking, the second marking and the third marking decrease in sequence. When the generation module 103 is used to process the initial stress marking of each syllable using the prosodic boundary and the tone of each syllable to obtain the stress marking of each syllable, it is specifically used to, for any syllable to be processed among all syllables, if the initial stress marking of the syllable to be processed is the first marking or the second marking, then the initial stress marking of the syllable to be processed is processed according to the tone of the syllable to be processed and the tone of each syllable before the syllable to be processed in the preset text to obtain the stress marking of the syllable to be processed; if the initial stress marking of the syllable to be processed is the third marking and the tone of the syllable to be processed is yangping or shangsheng, then the initial stress marking of the syllable to be processed is processed according to the valley value of the pitch curve of the syllable to be processed, the duration of the syllable to be processed and the prosodic boundary to obtain the stress marking of the syllable to be processed; traverse each syllable to obtain the stress marking of each syllable.

[0204] Optionally, the stress marking includes a first marking, a second marking and a third marking, and the stress levels represented by the first marking, the second marking and the third marking decrease in sequence. The generation module 103 is also used to treat all syllables with the stress marking as the first marking or the second marking as target syllables; for each target syllable, based on the peak values ​​of the tone and pitch curves of the adjacent syllables before and after the target syllable in the preset text, determine whether to modify the stress marking of the adjacent syllable to the stress marking of the target syllable.

[0205] Please refer to Figure 11 The speech synthesis device 200 includes a second acquisition module 201 and a synthesis module 202.

[0206] The second acquisition module 201 is used to acquire a reference text and linguistic features corresponding to the reference text.

[0207] The second acquisition module 201 is further configured to acquire reference audio data and acoustic features corresponding to the reference audio data.

[0208] The synthesis module 202 is used to input reference text, reference audio data, linguistic features and acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with accent effects corresponding to the reference text; wherein, the training samples used to train the speech synthesis model are obtained based on the aforementioned accent annotation generation method.

[0209] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described accent annotation generation device 100 and speech synthesis device 200 can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0210] Furthermore, the embodiment of the present invention also provides a computer device 300. Please refer to Figure 12 , the computer device 300 may include a memory 310 and a processor 320 .

[0211] The processor 320 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits for controlling the execution of the program of the accent annotation generation method and / or the speech synthesis method provided in the above-mentioned method embodiments.

[0212] The memory 310 may be a ROM or other type of static storage device that can store static information and instructions, a RAM or other type of dynamic storage device that can store information and instructions, or an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compact disc, laser disc, optical disc, digital versatile disc, Blu-ray disc, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto. The memory 310 may exist independently and be connected to the processor 320 via a communication bus. The memory 310 may also be integrated with the processor 320. The memory 310 is used to store machine executable instructions for executing the scheme of the present invention. The processor 320 is used to execute the machine executable instructions stored in the memory 310 to implement the above-mentioned method embodiment.

[0213] An embodiment of the present invention further provides a computer-readable storage medium containing a computer program. When the computer program is executed, it can be used to perform the method for generating accent markings and / or related operations in the speech synthesis method provided in the above method embodiment.

[0214] In summary, the embodiments of the present invention provide a method for generating accent annotations, a speech synthesis method, and related devices. First, based on the audio data corresponding to the preset text, the prosodic boundaries of the preset text and the tone and duration of each syllable in the preset text are obtained; then, the acoustic features of the audio data are extracted; finally, based on the prosodic boundaries, acoustic features, and the tone and duration of each syllable, the accent annotations for each syllable are generated. Because the embodiments of the present invention automatically generate accent annotations for each syllable using prosodic boundaries, acoustic features, and the tone and duration of each syllable, the time cost of accent annotations can be reduced, the accuracy of accent annotations can be improved, and the rhythm and expressiveness of the speech synthesized by the trained speech synthesis model can be guaranteed.

[0215] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by a person skilled in the art within the technical scope disclosed in the present invention should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be based on the scope of protection of the claims.< / pad>

Claims

1. A method for generating accent markings, characterized in that: The method comprises: Based on audio data corresponding to a preset text, obtaining a prosodic boundary of the preset text and a tone and duration of each syllable in the preset text; extracting acoustic features of the audio data; generating a stress mark for each syllable according to the prosodic boundary, the acoustic feature, and the tone and duration of each syllable; The audio data includes a plurality of speech frames, the acoustic features include a fundamental frequency of each of the speech frames, and the step of generating a stress mark for each syllable according to the prosodic boundary, the acoustic features, and the tone and duration of each syllable includes: For each syllable, determining at least one target speech frame corresponding to the syllable based on the duration of the syllable and the generation time of each speech frame, and generating a pitch curve of the syllable based on the fundamental frequencies of all the target speech frames, wherein the generation time of each target speech frame is within the duration of the syllable; generating an initial stress mark for each syllable according to the pitch curve of each syllable; The initial stress marking of each syllable is processed using the prosodic boundary and the tone of each syllable to obtain the stress marking of each syllable.

2. The method according to claim 1, wherein The step of generating an initial stress mark for each syllable according to the pitch curve of each syllable comprises: Performing statistical analysis on the pitch curves of all the syllables to obtain the pitch median and pitch standard deviation; For each syllable, an initial stress mark of the syllable is determined according to the pitch median, the pitch standard deviation, and the peak value of the pitch curve of the syllable.

3. The method according to claim 1, wherein The stress marking includes a first marking, a second marking and a third marking, and the stress levels represented by the first marking, the second marking and the third marking decrease in sequence; The step of processing the initial stress mark of each syllable by using the prosodic boundary and the tone of each syllable to obtain the stress mark of each syllable comprises: For any syllable to be processed among all the syllables, if the initial stress marking of the syllable to be processed is the first marking or the second marking, processing the initial stress marking of the syllable to be processed according to the tone of the syllable to be processed and the tone of each syllable in the preset text that precedes the syllable to be processed to obtain the stress marking of the syllable to be processed; If the initial stress marking of the syllable to be processed is the third marking and the tone of the syllable to be processed is Yangping or Shangsheng, processing the initial stress marking of the syllable to be processed according to the valley value of the pitch curve of the syllable to be processed, the duration of the syllable to be processed, and the prosodic boundary to obtain the stress marking of the syllable to be processed; Each of the syllables is traversed to obtain the stress mark of each of the syllables.

4. The method according to claim 1, wherein The accent marking includes a first marking, a second marking, and a third marking, wherein the accent levels represented by the first marking, the second marking, and the third marking decrease in sequence, and the method further includes: Among all the syllables, the syllables with the stress marked as the first mark or the second mark are all regarded as target syllables; For each target syllable, based on the peak values ​​of the tones and pitch curves of the adjacent syllables before and after the target syllable in the preset text, it is determined whether to modify the stress marking of the adjacent syllable to the stress marking of the target syllable.

5. A speech synthesis method, characterized in that: The method comprises: Obtaining a reference text and linguistic features corresponding to the reference text; Acquiring reference audio data and acoustic features corresponding to the reference audio; Inputting the reference text, the reference audio data, the linguistic features, and the acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with an accent effect corresponding to the reference text; The training samples used to train the speech synthesis model are obtained based on the method for generating accent annotations according to any one of claims 1 to 4.

6. A device for generating accent markings, characterized in that: The device comprises: A first acquisition module is configured to acquire, based on audio data corresponding to a preset text, a prosodic boundary of the preset text and a tone and duration of each syllable in the preset text; An extraction module, configured to extract acoustic features of the audio data; a generating module, configured to generate a stress mark for each syllable according to the prosodic boundary, the acoustic feature, and the tone and duration of each syllable; the audio data includes a plurality of speech frames, and the acoustic feature includes a fundamental frequency of each speech frame; The generation module is further configured to determine, for each syllable, at least one target speech frame corresponding to the syllable based on the duration of the syllable and the generation time of each speech frame, and generate a pitch curve of the syllable based on the fundamental frequencies of all the target speech frames, wherein the generation time of each target speech frame is within the duration of the syllable; generate an initial stress mark for each syllable based on the pitch curve of each syllable; and process the initial stress mark for each syllable using the prosodic boundary and the tone of each syllable to obtain a stress mark for each syllable.

7. A speech synthesis device, characterized in that: The device comprises: A second acquisition module is used to acquire a reference text and linguistic features corresponding to the reference text; The second acquisition module is further configured to acquire reference audio data and acoustic features corresponding to the reference audio data; a synthesis module, configured to input the reference text, the reference audio data, the linguistic features, and the acoustic features into a pre-trained speech synthesis model to obtain a synthesized speech with an accent effect corresponding to the reference text; The training samples used to train the speech synthesis model are obtained based on the method for generating accent annotations according to any one of claims 1 to 4.

8. A computer device, characterized in that: It includes a memory and a processor, the memory stores a computer program, and when the processor executes the computer program, it implements the method for generating accent markings according to any one of claims 1 to 4, and / or the method for speech synthesis according to claim 5.

9. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method for generating accent markings according to any one of claims 1 to 4 and / or the method for speech synthesis according to claim 5 are implemented.

Images