Prosody annotation method, device, computer device and storage medium

Abstract

The embodiments of the present application disclose a prosody annotation method, apparatus, computer equipment and storage medium, which belong to the field of computer technology. The method comprises: extracting semantic features of text information to obtain text semantic features of the text information, wherein the text semantic features represent the association between each two adjacent characters in the text information; extracting acoustic features of the speech information corresponding to the text information to obtain acoustic features of the speech information, wherein the acoustic features at least represent the pronunciation of each character in the speech information; and annotating the prosody label of each character in the speech information based on the text semantic features and the acoustic features. The method comprehensively considers text semantic features and acoustic features, relies on more information when predicting prosody labels, and improves the accuracy of the annotated prosody labels.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to a prosody annotation method, apparatus, computer device, and storage medium. Background Art

[0002] With the development of computer technology and speech synthesis technology, the method of synthesizing text into speech based on speech synthesis models is becoming more and more common. In order to improve the playback effect of the synthesized speech, it is necessary to accurately mark the prosodic label of each character in the speech information when training the speech synthesis model.

[0003] Currently, prosody prediction is typically performed directly on speech information to obtain a prosody label for each character in the speech information. However, since speech information only reflects the pronunciation characteristics of the characters, the prosody label prediction relies on limited information, resulting in poor accuracy. Summary of the Invention

[0004] The embodiments of the present application provide a prosody tagging method, apparatus, computer device, and storage medium, which improve the accuracy of the annotated prosody tags. The technical solution is as follows:

[0005] In one aspect, a prosody tagging method is provided, the method comprising:

[0006] Extracting semantic features from the text information to obtain text semantic features of the text information, wherein the text semantic features represent an association relationship between every two adjacent characters in the text information;

[0007] Extracting acoustic features from the speech information corresponding to the text information to obtain acoustic features of the speech information, where the acoustic features at least represent the pronunciation of each character in the speech information;

[0008] Based on the text semantic features and the acoustic features, a prosodic tag is labeled for each character in the speech information.

[0009] In another aspect, a prosody tagging device is provided, comprising:

[0010] A semantic feature acquisition module is used to extract semantic features from text information to obtain text semantic features of the text information, wherein the text semantic features represent the association relationship between every two adjacent characters in the text information;

[0011] an acoustic feature acquisition module, configured to extract acoustic features from the speech information corresponding to the text information to obtain acoustic features of the speech information, wherein the acoustic features at least represent the pronunciation of each character in the speech information;

[0012] A prosody tagging module is used to tag a prosody tag of each character in the speech information based on the text semantic features and the acoustic features.

[0013] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the prosody tagging method as described in the above aspects.

[0014] On the other hand, a computer-readable storage medium is provided, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the prosody tagging method as described in the above aspects.

[0015] In another aspect, a computer program product is provided. The computer program product stores at least one program code, and the at least one program code is loaded and executed by a processor to implement the operations performed in the prosody tagging method of the above embodiment.

[0016] The prosody tagging method, apparatus, computer device, and storage medium provided in the embodiments of the present application not only consider the acoustic features of the speech information itself when tagging each character in the speech information, but also consider the text semantic features of the text information corresponding to the speech information. The text semantic features can reflect the association relationship between each two adjacent characters. Therefore, when predicting the prosody tag, more information can be used, thereby improving the accuracy of the labeled prosody tag. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0018] Figure 1 This is a flow chart of a rhythm marking method provided by an embodiment of the present application;

[0019] Figure 2 This is a flow chart of a rhythm marking method provided by an embodiment of the present application;

[0020] Figure 3 is a schematic diagram of a rhythm tagging result provided by an embodiment of the present application;

[0021] Figure 4 is a schematic diagram of a rhythm annotation model provided in an embodiment of the present application;

[0022] Figure 5 This is a flow chart of a rhythm marking method provided by an embodiment of the present application;

[0023] Figure 6 is a schematic diagram of a rhythm annotation model provided in an embodiment of the present application;

[0024] Figure 7 Schematic diagram of the structure of a rhythm tagging device provided in an embodiment of the present application;

[0025] Figure 8 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0026] Figure 9 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0027] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below with reference to the accompanying drawings.

[0028] It is understood that the terms "first," "second," "third," "fourth," "fifth," "sixth," etc., used in this application may be used to describe various concepts herein, but unless otherwise specified, these concepts are not limited by these terms. These terms are merely used to distinguish one concept from another. For example, a first processing result may be referred to as a second processing result, and a second processing result may be referred to as a first processing result, without departing from the scope of this application.

[0029] As used herein, the terms "each," "plurality," "at least one," and "any" include one, two, or more, "plurality" includes two or more, "each" refers to each of the corresponding plurality, and "any" refers to any one of the plurality. For example, if a plurality of objects includes three objects, "each" refers to each of the three objects, and "any" refers to any one of the three objects, which can be the first, second, or third object.

[0030] An embodiment of the present application provides a prosody annotation method, the execution subject of which is a computer device. The computer device obtains text semantic features of text information and acoustic features of speech information corresponding to the text information, and annotates the prosody label of each character in the speech information based on the text semantic features and acoustic features to achieve prosody annotation.

[0031] Optionally, the computer device is a terminal, such as a mobile phone, desktop computer, laptop computer, tablet computer, smart TV, smart speaker, vehicle-mounted terminal, smart robot, etc. Alternatively, the computer device is a server, which can be a single server, a server cluster consisting of multiple servers, or a cloud computing server center.

[0032] By using the prosody labeling method provided in the embodiment of the present application, after obtaining speech information labeled with prosody labels, a speech synthesis model can be trained based on the speech information, and then the trained speech synthesis model can be applied to a variety of speech synthesis scenarios, such as voice mailboxes, telephone banks, station broadcast systems, elevator broadcast systems, voice notification systems, hospital call systems, etc. The embodiment of the present application does not limit the speech synthesis scenarios.

[0033] Figure 1 This is a flow chart of a rhythm tagging method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device, see Figure 1 , the method comprising:

[0034] 101. The computer device extracts semantic features from the text information to obtain text semantic features of the text information.

[0035] The text information is any text information, and the text semantic feature represents the association relationship between every two adjacent characters in the text information, that is, the text semantic feature can represent the meaning of the text information.

[0036] 102. The computer device extracts acoustic features of the voice information corresponding to the text information to obtain acoustic features of the voice information.

[0037] The voice information and the text information correspond to each other, i.e., the content of the voice information is consistent with the content of the text information, and the acoustic features of the voice information at least represent the pronunciation of each character in the voice information. Optionally, the acoustic features include at least one of a sound intensity feature, a pitch feature, a pronunciation duration feature, a silence duration feature, or a pinyin feature.

[0038] 103. The computer device annotates the prosodic label of each character in the speech information based on the semantic features and acoustic features of the text.

[0039] When labeling prosodic information, acoustic features can reflect some characteristics of speech pronunciation, while text semantic features can reflect some characteristics of the corresponding textual content semantics. Taking these two characteristics into consideration, the resulting prosodic labels are more accurate. A prosodic label at least indicates whether there is a pause between one character and the next, indicating the prosodic boundary of that character.

[0040] The prosody tagging method provided in the embodiment of the present application not only considers the acoustic features of the speech information itself when tagging each character in the speech information, but also considers the text semantic features of the text information corresponding to the speech information. The text semantic features can reflect the association relationship between each two adjacent characters. Therefore, when predicting the prosody label, more information can be used, thereby improving the accuracy of the labeled prosody label.

[0041] Figure 2 This is a flow chart of a rhythm tagging method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device, see Figure 2 , the method comprising:

[0042] 201. A computer device obtains text information and voice information corresponding to the text information.

[0043] The text information corresponds to the voice information, that is, each character in the text information corresponds to each character in the voice information.

[0044] In one possible implementation, the text information and the voice information are sent to the computer device by another device; or, the text information and the voice information are collected by the computer device. Optionally, the computer device provides text information to the user, and the user sends a voice message corresponding to the text information based on the text information, and the computer device collects the voice information, thereby obtaining the text information and voice information. Alternatively, the computer device collects the voice information sent by the user and then recognizes the corresponding text information based on the voice information, or the user inputs the corresponding text information based on the voice information. The embodiment of the present application does not limit the method for obtaining the text information and voice information.

[0045] It should be noted that the embodiments of the present application do not limit the content, number of characters, etc. of text information and voice information.

[0046] 202. The computer device extracts semantic features from the text information to obtain text semantic features of the text information.

[0047] The text semantic feature represents the association relationship between every two adjacent characters in the text information, that is, the text semantic feature can represent the meaning of the text information.

[0048] In one possible implementation, a computer device extracts semantic features from multiple characters in a text message to obtain a character semantic feature for each character in the text message. For each character, when extracting the character semantic feature, not only are features reflecting the meaning of the character itself extracted, but also features reflecting the meaning of the character within the entire text message are extracted in conjunction with the context of the character in the text message. Therefore, the character semantic feature represents both the semantics of the character itself and the semantics of the character within the text message. The character semantic features of these multiple characters are then fused to obtain a text semantic feature.

[0049] Optionally, the computer device concatenates the character semantic features of multiple characters to obtain the text semantic feature; or, the computer device performs weighted processing on the character semantic features of the multiple characters based on the importance of the multiple characters in the text information to obtain the text semantic feature.

[0050] It should be noted that the embodiment of the present application is only described using the above-mentioned method of extracting text semantic features as an example. In another embodiment, the computer device can obtain text semantic features in other ways. The embodiment of the present application does not limit the specific method of extracting text semantic features.

[0051] 203. The computer device extracts acoustic features of the speech information to obtain acoustic features of the speech information.

[0052] Among them, the acoustic feature represents the pronunciation of the speech information, and the acoustic feature can reflect the characteristics of the speech information in terms of pitch, sound intensity, pronunciation duration, etc.

[0053] In one possible implementation, since the entire speech information contains not only the syllable segments of each character, but also some silent segments other than the syllable segments, and these silent segments will affect the accuracy of the extracted acoustic features, when the computer device extracts acoustic features from the speech information, it first identifies the syllable segment corresponding to each character in the speech information, where the syllable segment includes the pronunciation of the corresponding character but does not include silence or other noise other than the pronunciation. Then, acoustic feature extraction is performed based on each syllable segment to obtain the acoustic features of each character, and the acoustic features of multiple characters are concatenated to obtain the acoustic features of the speech information.

[0054] In the embodiment of the present application, the acoustic features include at least one of a sound intensity feature, a pitch feature, a pronunciation duration feature, a silence duration feature, or a pinyin feature. The following describes the extraction process for each of the features:

[0055] (1) Extracting sound intensity features:

[0056] The computer device extracts the sound intensity feature of each syllable segment to obtain the sound intensity feature of each character. The sound intensity refers to the loudness of the sound.

[0057] In one possible implementation, since the pitch of the same character changes during pronunciation, and the pronunciation duration of different characters also varies, in order to prevent the different pronunciation durations from affecting the extracted sound intensity features, each syllable segment is divided into multiple sub-segments, with each two sub-segments corresponding to the same duration. Sound intensity features are then extracted from each of the multiple sub-segments to obtain the sound intensity sub-features corresponding to each sub-segment. The multiple sound intensity sub-features are averaged to obtain the sound intensity features of the character corresponding to the syllable segment. That is, by averaging, the sound intensity features corresponding to each character correspond to the same duration.

[0058] Optionally, based on the first reference duration, the syllable segment is divided into multiple sub-segments, so that the duration of each sub-segment is the first reference duration.

[0059] Optionally, for multiple syllable segments, the first reference durations used when dividing the multiple syllable segments into sub-segments may be the same or different.

[0060] (2) Extracting pitch features:

[0061] The computer device extracts the pitch feature of each syllable segment to obtain the pitch feature of each character. Pitch refers to the highness of the tone.

[0062] In one possible implementation, since the pitch of the same character changes during pronunciation, and the pronunciation duration of different characters also varies, in order to avoid the influence of different pronunciation durations on the extracted pitch features, for each syllable segment, the syllable segment is divided into multiple sub-segments, and the duration corresponding to every two sub-segments is the same. Then, pitch features are extracted from the multiple sub-segments to obtain the pitch sub-features corresponding to each sub-segment. The multiple pitch sub-features are averaged to obtain the pitch features of the character corresponding to the syllable segment. That is, by averaging, the pitch features corresponding to each character correspond to the same duration.

[0063] Optionally, based on the second reference duration, the syllable segment is divided into multiple sub-segments, so that the duration of each sub-segment is the second reference duration. The second reference duration may be the same as or different from the first reference duration.

[0064] Optionally, for multiple syllable segments, the second reference durations used as a basis for dividing the multiple syllable segments into sub-segments may be the same or different.

[0065] (3) Extracting pronunciation duration features:

[0066] The computer device extracts features based on the duration of each syllable segment to obtain a pronunciation duration feature of each character. The pronunciation duration feature indicates the duration of the character pronunciation.

[0067] (4) Extracting silence duration features:

[0068] The computer device extracts features based on the duration of the interval between each two adjacent syllable segments to obtain a silence duration feature of the character corresponding to the first audio segment in each two-syllable segment. The silence duration feature represents the duration of the interval between a character and the next adjacent character.

[0069] Optionally, the computer device determines a silent segment between each two syllable segments based on the voice information, and the duration of the silent segment is the interval duration between each two syllable segments. If there is no silent segment between each two syllable segments, the interval duration between the two syllable segments can be determined to be 0.

[0070] (5) Extracting Pinyin features:

[0071] The computer device extracts pinyin features based on each syllable segment to obtain a pinyin feature for each character. Due to the existence of polyphonetic characters, extracting the pinyin features of characters directly based on text information may result in the determined pinyin feature not being the pronunciation of the character in the text information. Therefore, the pinyin feature of the character is determined directly based on the pronunciation of the character in the syllable segment.

[0072] It should be noted that the embodiments of the present application are only described using the above-mentioned acoustic features as examples. In another embodiment, other acoustic features may also be included, and the present application does not impose any restrictions on this.

[0073] In one possible implementation, before extracting acoustic features, the computer device first denoises the speech information to prevent noise in the speech information from affecting the acoustic feature extraction, and then performs acoustic feature extraction on the denoised speech information. For example, the speech information can be denoised using an adaptive filter, spectral subtraction, or other denoising algorithm. This embodiment of the present application does not limit the denoising method.

[0074] It should be noted that the embodiment of the present application is only described by taking the example of first executing 202 and then executing step 203. In another embodiment, the computer device can first execute step 203 and then execute step 202, or execute step 202 and step 203 at the same time. The embodiment of the present application does not limit the order of execution of the steps.

[0075] 204. The computer device determines punctuation information of the text information based on the text information.

[0076] When there are multiple sentences in a voice message, the pauses between sentences are also very important. To identify the pauses between two adjacent sentences, the computer device obtains the punctuation information in the text information, which includes the position of the punctuation in the text information and the punctuation type. Among them, the position of the punctuation can be determined by the characters before or after the punctuation, and the punctuation types include commas, periods, colons, semicolons, etc.

[0077] 205. The computer device labels the prosody tags of each character in the voice message based on the text semantic features, acoustic features, and punctuation information.

[0078] The computer device considers the text semantic features, acoustic features, and punctuation information to label the prosody tags of each character in the voice message.

[0079] Optionally, the prosody tag includes a first tag and a second tag. The first tag indicates that there is no pause between the character and the next character of the character, and the second tag indicates that there is a pause between the character and the next character of the character, as well as the pause duration between the character and the previous character of the character.

[0080] Optionally, according to the pause duration, the second tag is further divided into multiple sub-tags. For example, the pause duration between 0.05 seconds and 0.1 seconds corresponds to the same sub-tag, and the pause duration between 0.11 seconds and 0.15 seconds corresponds to the same sub-tag. For example, 0, 1, 2, 3, and 4 are used to represent the prosody tags, 0 represents the first tag, 1, 2, 3, and 4 represent the second tag, and the pause durations corresponding to 1, 2, 3, and 4 are getting longer.

[0081] For example, see Figure 3 , for the text information "You can also hand in your homework and take exams.", the corresponding prosody tags are labeled for each character in the voice message corresponding to the text information. It can be seen from Figure 3 that when two adjacent characters can form a word, the prosody tag between these two characters is "0", that is, there is no pause between these two characters. The prosody tag between two words is not "0", which means there is a pause between these two words. There is a comma between two sentences. At this time, the prosody tag corresponding to the character "work" before the comma is "3", and the pause duration after "work" is longer than the pause duration between two adjacent words, that is, the existence of punctuation has a certain impact on the prosody tag.

[0082] For another example, see the following features other than the text semantic features when determining the prosody tags given in the following example:

[0083] text: [one, same, can, also, hand in, homework, and, take, exams]

[0084] Pinyin: [yi, yang, ke, yi, jiao, zuo, ye, he, jin, xing, kao, shi]

[0085] Phons: [[y, i], [y, ang], [k, e], [y, i], [j, iao],...]

[0086] Pitch: [[220, 221, 225, 226], [234, 256,...],...]

[0087] Energy: [[20, 21, 25, 26], [34, 56,...],...]

[0088] Dur: [2, 3, 21, 31,...]

[0089] Sil_dur: [0, 0, 0,..., 3]

[0090] Punc: [,,,,,, ,,,,,, 。]

[0091] Prosody: [0, 1, 0, 2, 0, 0, 3, 1, 0, 1, 0, 4]

[0092] Among them, text represents the text information corresponding to the speech information, Pinyin represents the Pinyin corresponding to the characters, Phons represents the initials and finals of the characters, Pitch represents the pitch characteristics of the characters (220, 221, 225, 226 represent the pitch sub - characteristics of the sub - segments corresponding to "yi"), Energy represents the intensity characteristics of the characters (20, 21, 25, 26 represent the intensity sub - characteristics of the sub - segments corresponding to "yi"), Dur represents the pronunciation duration characteristics of the characters, Sil_dur represents the silent duration characteristics after the characters, Punc represents the punctuation information, and Prosody represents the prosody label.

[0093] It should be noted that step 204 is an optional step. In another embodiment, step 204 may not be executed. Then, in the case of not executing step 204, the computer device directly labels the prosody label of each character in the speech information based on the text semantic characteristics and acoustic characteristics.

[0094] The above embodiment describes the process of annotating prosodic labels. In one possible implementation, after annotating each character in the speech information with a prosodic label, the text information, speech information, and the prosodic label of each character in the speech information can be determined as training data for a speech synthesis model. The speech synthesis model is then invoked to perform speech synthesis on the text information to obtain synthesized speech information. The speech synthesis model is then trained based on the synthesized speech information, the speech information, and the prosodic label of each character in the speech information. Specifically, the pauses of each character in the synthesized speech information are compared with the prosodic label of each character in the speech information to determine whether each character in the synthesized speech information matches the corresponding prosodic label. The model parameters of the speech synthesis model are then adjusted based on the comparison results.

[0095] Another point that needs to be explained is that the embodiment of the present application is only described by taking the prosodic tag of each character in the voice information as an example. In another embodiment, the computer device can also mark the prosodic tag of each character in the text information.

[0096] The prosodic tagging method provided in the embodiments of the present application not only considers the acoustic features of the speech information itself when tagging each character in speech information, but also the textual semantic features of the text information corresponding to the speech information. These textual semantic features can reflect the relationship between each two adjacent characters. Therefore, when predicting the prosodic tag, more information can be used, thereby improving the accuracy of the annotated prosodic tag. In addition, punctuation information is also considered, which can further improve the accuracy of the annotated prosodic tag.

[0097] Moreover, in the embodiment of the present application, multiple acoustic features such as pitch features, sound intensity features, pronunciation duration features, silence duration features, and pinyin features are taken into consideration. When marking prosodic labels, multiple acoustic features are used as a basis, which further improves the accuracy of the marked prosodic labels.

[0098] Moreover, compared with the manual prosodic labeling method in the related art, the prosodic labeling method provided in the embodiment of the present application is not affected by individual differences among labelers, and can adopt a unified labeling standard for labeling, thereby improving the stability of prosodic labeling.

[0099] In another embodiment, the computer device can label the prosody tags by calling the prosody labeling model. Figure 4 The prosody annotation model includes a text processing network 401, a speech processing network 402, and a prosody annotation network 403. The text processing network 401 and the speech processing network 402 are connected to the prosody annotation network 403 respectively.

[0100] Figure 5This is a flow chart of a rhythm tagging method provided by an embodiment of the present application. The execution subject of the embodiment of the present application is a computer device. Figure 5 , the method comprising:

[0101] 501. The computer device calls a text processing network to extract semantic features of the text information to obtain text semantic features of the text information.

[0102] The computer device inputs the text information into the text processing network to obtain the text semantic features.

[0103] In one possible implementation, the text processing network is a BERT (Bidirectional Encoder Representations from Transformers) network. The BERT network contains pre-trained information corresponding to characters. Because the BERT network uses a multi-layer Transformer to process text bidirectionally, it can accurately learn the contextual relationships between multiple characters in the text information. And because the BERT network has the function of recognizing each character in the text information, there is no need to use a large amount of sample data to train the BERT network in the subsequent training process. Alternatively, the text processing network is other networks that have the function of extracting semantic features.

[0104] 502. The computer device calls the speech processing network to extract acoustic features of the speech information to obtain acoustic features of the speech information.

[0105] In one possible implementation, since multiple different types of acoustic features need to be extracted and the extraction methods for these different types of acoustic features are different, in order to accurately extract each acoustic feature, the speech processing network includes multiple sub-networks, each of which is used to extract an acoustic feature.

[0106] Optionally, the computer device calls the first sub-network to extract the sound intensity feature of each syllable segment to obtain the sound intensity feature of each character; calls the second sub-network to extract the pitch feature of each syllable segment to obtain the pitch feature of each character; calls the third sub-network to extract features based on the duration of each syllable segment to obtain the pronunciation duration feature of each character; calls the fourth sub-network to extract features based on the interval duration between each two adjacent syllable segments to obtain the silence duration feature of the character corresponding to the first audio segment in each of the two syllable segments; calls the fifth sub-network to extract pinyin features based on each syllable segment to obtain the pinyin feature of each character.

[0107] 503. The computer device calls the prosody tagging network to tag the prosody label of each character in the speech information based on the semantic features and acoustic features of the text.

[0108] The computer device inputs the acquired text semantic features and acoustic features into the prosody annotation network respectively, predicts the prosody label of each character in the speech information based on the prosody annotation network, and annotates the corresponding prosody label for each character.

[0109] In one possible implementation, the prosodic annotation network includes a recognition network and an annotation network. The recognition network is used to identify semantic and acoustic features of the text and determine the prosodic label corresponding to each character, and the annotation network is used to label each character with the corresponding prosodic label. Optionally, the recognition network is an LSTM (Long Short Term Memory) network, a Bi-LSTM (Bidirectional Long Short Term Memory) network, an HMM (Hidden Markov Model), or other network.

[0110] For example, see Figure 6 Through the text processing network, the semantic features of the text information "Karp accompanies his grandson to play on the slide" are extracted to obtain the text semantic features. Through the speech processing network, the speech information is processed to obtain acoustic features such as pronunciation duration features, pitch features, and sound intensity features. The text semantic features and the acoustic features of each character are input into the Bi-LSTM network. For multiple characters, they are processed through the forward long short-term memory layer (Forward-LSTM) and the backward long short-term memory layer (Backward-LSTM) respectively to obtain the prosodic label of each character.

[0111] It should be noted that the embodiment of this application only introduces the process of using the prosody annotation model. In another embodiment, before using the prosody annotation model, it is necessary to first train the prosody annotation model. That is, the computer device obtains sample text information, sample voice information, and sample prosody labels for each character in the sample voice information, inputs the sample text information and sample voice information into the prosody annotation model, obtains the predicted prosody label for each character, and adjusts the model parameters of the prosody annotation model based on the differences between the multiple predicted prosody labels and the sample prosody labels. The prosody annotation model is iteratively trained multiple times until a usable prosody annotation model is obtained.

[0112] The prosody annotation method provided in the embodiment of the present application, when calling the prosody annotation model to annotate a prosody label for each character in the speech information, not only considers the acoustic features of the speech information itself, but also considers the text semantic features of the text information corresponding to the speech information. The text semantic features can reflect the association relationship between each two adjacent characters. Therefore, when predicting the prosody label, it can be based on more information, thereby improving the accuracy of the annotated prosody label.

[0113] Figure 7 This is a schematic diagram of the structure of a rhythm tagging device provided by an embodiment of the present application. Figure 7 , the device comprises:

[0114] Semantic feature acquisition module 701, used to extract semantic features from text information to obtain text semantic features of the text information, wherein the text semantic features represent the association relationship between every two adjacent characters in the text information;

[0115] Acoustic feature acquisition module 702, configured to extract acoustic features from the speech information corresponding to the text information to obtain acoustic features of the speech information, wherein the acoustic features at least represent the pronunciation of each character in the speech information;

[0116] The prosody tagging module 703 is configured to tag a prosody tag of each character in the speech information based on the text semantic features and the acoustic features.

[0117] The device provided in the embodiment of the present application, when labeling a prosodic label for each character in speech information, not only considers the acoustic features of the speech information itself, but also considers the text semantic features of the text information corresponding to the speech information. The text semantic features can reflect the association relationship between each two adjacent characters. Therefore, when predicting the prosodic label, it can be based on more information, thereby improving the accuracy of the labeled prosodic label.

[0118] In one possible implementation, the acoustic feature acquisition module 702 includes:

[0119] a segment recognition unit, configured to recognize a syllable segment corresponding to each character in the voice information;

[0120] An acoustic feature acquisition unit, configured to extract acoustic features based on each syllable segment to obtain acoustic features of each character;

[0121] The acoustic feature concatenation unit is used to concatenate the acoustic features of multiple characters to obtain the acoustic features of the speech information.

[0122] In one possible implementation, the acoustic feature acquisition unit is configured to:

[0123] Extract the sound intensity feature of each syllable segment to obtain the sound intensity feature of each character;

[0124] Extract the pitch feature of each syllable segment to obtain the pitch feature of each character;

[0125] Perform feature extraction based on the duration of each syllable segment to obtain the pronunciation duration feature of each character;

[0126] Extract features based on the interval duration between each two adjacent syllable segments to obtain a silence duration feature of the character corresponding to the first audio segment in each of the two syllable segments;

[0127] Pinyin features are extracted based on each syllable segment to obtain the pinyin features of each character.

[0128] In one possible implementation, the acoustic feature acquisition unit is configured to:

[0129] For each of the syllable segments:

[0130] Divide the syllable segment into multiple sub-segments, and each two sub-segments have the same duration;

[0131] Extracting sound intensity features from the plurality of sub-segments respectively to obtain sound intensity sub-features corresponding to each sub-segment;

[0132] The average of the multiple sound intensity sub-features is calculated to obtain the sound intensity feature of the character corresponding to the syllable segment.

[0133] In one possible implementation, the acoustic feature acquisition unit is configured to:

[0134] For each of the syllable segments:

[0135] Dividing the syllable segment into multiple sub-segments, where every two sub-segments have the same duration;

[0136] Extracting pitch features of the plurality of sub-segments respectively to obtain pitch sub-features corresponding to each sub-segment;

[0137] The average of the multiple pitch sub-features is calculated to obtain the pitch feature of the character corresponding to the syllable segment.

[0138] In one possible implementation, the device further includes:

[0139] a punctuation information acquisition module, configured to determine punctuation information of the text information based on the text information;

[0140] The prosody annotation module is used to:

[0141] A prosodic tag is assigned to each character in the speech information based on the text semantic feature, the acoustic feature, and the punctuation information.

[0142] In one possible implementation, the semantic feature acquisition module is used to:

[0143] Extracting semantic features of multiple characters in the text information respectively to obtain a character semantic feature of each character in the text information;

[0144] The character semantic features of the multiple characters are fused to obtain the text semantic feature.

[0145] In one possible implementation, the prosody annotation model includes a text processing network, a speech processing network, and a prosody annotation network.

[0146] The text processing network is used to extract semantic features from the text information to obtain the semantic features of the text;

[0147] The speech processing network is used to extract acoustic features from the speech information to obtain acoustic features of the speech information;

[0148] The prosody tagging network is used to tag the prosody label of each character in the speech information based on the text semantic feature and the acoustic feature.

[0149] In one possible implementation, the device further includes:

[0150] a training data determination module, configured to determine the text information, the voice information, and the prosodic label of each character in the voice information as training data for a speech synthesis model;

[0151] A speech synthesis module is used to call the speech synthesis model to perform speech synthesis on the text information to obtain synthesized speech information;

[0152] A model training module is used to train the speech synthesis model based on the synthesized speech information, the speech information and the prosodic label of each character in the speech information.

[0153] In a possible implementation, the prosody tagging module is further configured to tag a prosody tag of each character in the text information based on the text semantic feature and the acoustic feature.

[0154] In one possible implementation, the prosody tag includes a first tag and a second tag, the first tag indicating that there is no pause between the character and the next character, the second tag indicating that there is a pause between the character and the next character, and the length of the pause between the character and the next character.

[0155] An embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the prosody tagging method of the above embodiment.

[0156] Optionally, the computer device is provided as a terminal. Figure 8 The figure is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. Terminal 800 may be a portable mobile terminal, such as a smartphone, a tablet computer, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, or a desktop computer. Terminal 800 may also be referred to as user equipment, a portable terminal, a laptop terminal, a desktop terminal, or other similar names.

[0157] The terminal 800 includes a processor 801 and a memory 802 .

[0158] The processor 801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 801 may be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), or PLA (Programmable Logic Array). The processor 801 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0159] The memory 802 may include one or more computer-readable storage media, which may be non-transitory. The memory 802 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 802 is used to store at least one program code, which is executed by the processor 801 to implement the prosody tagging method provided in the method embodiment of the present application.

[0160] In some embodiments, terminal 800 may optionally include a peripheral device interface 803 and at least one peripheral device. The processor 801, memory 802, and peripheral device interface 803 may be connected via a bus or signal lines. Each peripheral device may be connected to peripheral device interface 803 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 804, a display screen 805, a camera assembly 806, an audio circuit 807, a positioning assembly 808, and a power supply 809.

[0161] The peripheral device interface 803 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 801 and the memory 802. In some embodiments, the processor 801, the memory 802, and the peripheral device interface 803 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 801, the memory 802, and the peripheral device interface 803 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0162] The radio frequency circuit 804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 804 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 804 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 804 includes an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, and the like. The radio frequency circuit 804 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 20G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the radio frequency circuit 804 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0163] The display screen 805 is used to display a user interface (UI). This UI may include graphics, text, icons, videos, or any combination thereof. When the display screen 805 is a touch screen display, it is also capable of collecting touch signals on or above the surface of the display screen 805. These touch signals can be input as control signals to the processor 801 for processing. In this case, the display screen 805 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there can be one display screen 805, disposed on the front panel of the terminal 800; in other embodiments, there can be at least two display screens 805, disposed on different surfaces of the terminal 800 or in a foldable design; in still other embodiments, the display screen 805 can be a flexible display, disposed on a curved or foldable surface of the terminal 800. Furthermore, the display screen 805 can be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. The display screen 805 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0164] The camera assembly 806 is used to capture images or videos. Optionally, the camera assembly 806 includes a front camera and a rear camera. The front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and VR (Virtual Reality) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 806 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. The dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0165] The audio circuit 807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into the processor 801 for processing, or input into the radio frequency circuit 804 to achieve voice communication. For the purpose of stereo sound collection or noise reduction, there may be multiple microphones, each located in different parts of the terminal 800. The microphone may also be an array microphone or an omnidirectional collection microphone. The speaker is used to convert electrical signals from the processor 801 or the radio frequency circuit 804 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 807 may also include a headphone jack.

[0166] Positioning component 808 is used to locate the current geographic location of terminal 800 to implement navigation or LBS (Location Based Service). Positioning component 808 can be based on the US GPS (Global Positioning System), China's Beidou system, Russia's Greninja positioning system, or the European Union's Galileo positioning system.

[0167] Power supply 809 is used to power various components in terminal 800. Power supply 809 can be AC power, DC power, a disposable battery, or a rechargeable battery. When power supply 809 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, while a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0168] In some embodiments, the terminal 800 further includes one or more sensors 180 , including but not limited to: an acceleration sensor 811 , a gyroscope sensor 812 , a pressure sensor 813 , a fingerprint sensor 814 , an optical sensor 815 , and a proximity sensor 816 .

[0169] The accelerometer 811 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal 80. For example, the accelerometer 811 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 801 can control the display screen 805 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 811. The accelerometer 811 can also be used to collect game or user motion data.

[0170] The gyroscope sensor 812 can detect the orientation and rotation angle of the terminal 800. It can work in conjunction with the accelerometer 811 to collect the user's 3D movements of the terminal 800. Based on the data collected by the gyroscope sensor 812, the processor 801 can implement the following functions: motion sensing (for example, changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0171] The pressure sensor 813 can be provided on the side frame of the terminal 800 and / or below the display screen 802. When the pressure sensor 813 is provided on the side frame of the terminal 800, it can detect the user's gripping signal of the terminal 800, and the processor 801 can perform left and right hand recognition or shortcut operations based on the gripping signal collected by the pressure sensor 813. When the pressure sensor 813 is provided below the display screen 802, the processor 801 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 805. Operable controls include at least one of a button control, a scroll bar control, an icon control, and a menu control.

[0172] The fingerprint sensor 814 is used to collect the user's fingerprint. The processor 801 identifies the user's identity based on the fingerprint collected by the fingerprint sensor 814, or the fingerprint sensor 814 identifies the user's identity based on the collected fingerprint. When the user's identity is recognized as a trusted identity, the processor 801 authorizes the user to perform relevant sensitive operations, such as unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 814 can be set on the front, back, or side of the terminal 800. When a physical button or manufacturer logo is provided on the terminal 800, the fingerprint sensor 814 can be integrated with the physical button or manufacturer logo.

[0173] The optical sensor 815 is used to detect ambient light intensity. In one embodiment, the processor 801 can control the display brightness of the display screen 805 based on the ambient light intensity detected by the optical sensor 815. Specifically, when the ambient light intensity is high, the display brightness of the display screen 805 is increased; when the ambient light intensity is low, the display brightness of the display screen 805 is decreased. In another embodiment, the processor 801 can also dynamically adjust the shooting parameters of the camera assembly 806 based on the ambient light intensity detected by the optical sensor 815.

[0174] Proximity sensor 816, also known as a distance sensor, is disposed on the front panel of terminal 800. Proximity sensor 816 is used to detect the distance between the user and the front of terminal 800. In one embodiment, when proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually decreasing, processor 801 controls display screen 805 to switch from the screen-on state to the screen-off state. When proximity sensor 816 detects that the distance between the user and the front of terminal 800 is gradually increasing, processor 801 controls display screen 805 to switch from the screen-off state to the screen-on state.

[0175] Those skilled in the art will understand that Figure 8 The structure shown in the figure does not constitute a limitation on the terminal 800, and the terminal 800 may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0176] Optionally, the computer device is provided as a server. Figure 9 This is a schematic diagram of the structure of a server according to an exemplary embodiment. The server 900 may vary significantly due to different configurations or performance, and may include one or more processors (Central Processing Units, CPUs) 901 and one or more memories 902. The memories 902 store at least one program code, which is loaded and executed by the processor 901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and input and output interfaces for input and output. The server may also include other components for implementing device functions, which will not be described in detail here.

[0177] An embodiment of the present application further provides a computer-readable storage medium, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the prosody tagging method of the above embodiment.

[0178] An embodiment of the present application further provides a computer program product, in which at least one program code is stored. The at least one program code is loaded and executed by a processor to implement the operations performed in the prosody tagging method of the above embodiment.

[0179] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0180] The above are only optional embodiments of the embodiments of the present application and are not intended to limit the embodiments of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the scope of protection of this application.

Claims

1. A rhythm tagging method, characterized in that: The method comprises: Extracting semantic features from the text information to obtain text semantic features of the text information, wherein the text semantic features represent an association relationship between every two adjacent characters in the text information; Extracting acoustic features from the speech information corresponding to the text information to obtain acoustic features of the speech information, where the acoustic features at least represent the pronunciation of each character in the speech information, the content of the speech information being consistent with the content of the text information, and the acoustic features including at least one of a sound intensity feature, a pitch feature, a pronunciation duration feature, a silence duration feature, or a pinyin feature; Determining punctuation information of the text information based on the text information, the punctuation information including a position of the punctuation in the text information and a punctuation type; A prosodic tag is added to each character in the speech information based on the punctuation information, the text semantic features, and the acoustic features.

2. The method according to claim 1, characterized in that The extracting acoustic features of the voice information corresponding to the text information to obtain the acoustic features of the voice information includes: Identifying a syllable segment corresponding to each character in the voice information; Extracting acoustic features based on each syllable segment to obtain acoustic features of each character; The acoustic features of the plurality of characters are concatenated to obtain the acoustic features of the speech information.

3. The method according to claim 2, characterized in that The acoustic feature extraction is performed based on each syllable segment to obtain the acoustic feature of each character, including at least one of the following: Extracting the sound intensity feature of each syllable segment to obtain the sound intensity feature of each character; Extracting pitch features of each syllable segment to obtain pitch features of each character; Performing feature extraction based on the duration of each syllable segment to obtain a pronunciation duration feature of each character; Extracting features based on the interval duration between each two adjacent syllable segments to obtain a silence duration feature of the character corresponding to the first audio segment in each of the two syllable segments; Pinyin features are extracted based on each of the syllable segments to obtain the pinyin features of each of the characters.

4. The method according to claim 3, characterized in that The extracting the sound intensity feature of each syllable segment to obtain the sound intensity feature of each character includes: For each of the syllable segments: Dividing the syllable segment into multiple sub-segments, where every two sub-segments have the same duration; Extracting sound intensity features from each of the sub-segments to obtain a sound intensity sub-feature corresponding to each of the sub-segments; An average of the plurality of sound intensity sub-features is calculated to obtain the sound intensity feature of the character corresponding to the syllable segment.

5. The method according to claim 3, characterized in that The step of extracting the pitch feature of each syllable segment to obtain the pitch feature of each character includes: For each of the syllable segments: Dividing the syllable segment into multiple sub-segments, where every two sub-segments have the same duration; Extracting pitch features from each of the sub-segments to obtain a pitch sub-feature corresponding to each of the sub-segments; An average of the plurality of pitch sub-features is calculated to obtain the pitch feature of the character corresponding to the syllable segment.

6. The method according to claim 1, characterized in that The extracting semantic features of the text information to obtain the text semantic features of the text information includes: Extracting semantic features of the plurality of characters in the text information respectively to obtain a character semantic feature of each character in the text information; The character semantic features of the multiple characters are fused to obtain the text semantic feature.

7. The method according to claim 1, characterized in that The prosody annotation model includes a text processing network, a speech processing network, and a prosody annotation network. The text processing network is used to extract semantic features from the text information to obtain the text semantic features; The speech processing network is used to extract acoustic features from the speech information to obtain acoustic features of the speech information; The prosody tagging network is used to tag the prosody label of each character in the speech information based on the text semantic features and the acoustic features.

8. The method according to claim 1, characterized in that The method further comprises: Determining the text information, the voice information, and the prosodic label of each character in the voice information as training data for a speech synthesis model; Calling the speech synthesis model to perform speech synthesis on the text information to obtain synthesized speech information; The speech synthesis model is trained based on the synthesized speech information, the speech information, and a prosodic label of each character in the speech information.

9. The method according to claim 1, characterized in that The method further comprises: A prosodic tag is assigned to each character in the text information based on the text semantic feature and the acoustic feature.

10. The method according to any one of claims 1 to 9, characterized in that The prosody tag includes a first tag and a second tag, wherein the first tag indicates that there is no pause between the character and the next character, and the second tag indicates that there is a pause between the character and the next character, and the length of the pause between the character and the next character.

11. A rhythm tagging device, characterized in that: The device comprises: A semantic feature acquisition module is used to extract semantic features from text information to obtain text semantic features of the text information, wherein the text semantic features represent the association relationship between every two adjacent characters in the text information; an acoustic feature acquisition module, configured to extract acoustic features from the speech information corresponding to the text information to obtain acoustic features of the speech information, wherein the acoustic features at least represent the pronunciation of each character in the speech information, the content of the speech information being consistent with the content of the text information, and the acoustic features including at least one of a sound intensity feature, a pitch feature, a pronunciation duration feature, a silence duration feature, or a pinyin feature; a punctuation information acquisition module, configured to determine punctuation information of the text information based on the text information, the punctuation information including the position of the punctuation in the text information and the type of the punctuation; A prosody tagging module is used to tag a prosody tag of each character in the speech information based on the punctuation information, the text semantic features and the acoustic features.

12. A computer device, characterized in that: The computer device includes a processor and a memory, wherein the memory stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the prosody tagging method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one program code, and the at least one program code is loaded and executed by the processor to implement the operations performed in the prosody tagging method according to any one of claims 1 to 10.

Images