A method for identifying the prosody of a text, a device and a method for training a model

Abstract

The application provides a prosody recognition method and device of text and a method for training a model, so that the prosody of recorded audio and the prosody label in the annotated text are accurately matched. The method comprises: obtaining first audio recorded by a first speaker according to first text; determining a first position in the first text corresponding to a pause time period of the audio according to the first audio and the first text; inputting the first text into a first neural network model to predict a first intermediate prosody label, obtaining second text, and the second text comprising the first intermediate prosody label and the first text; determining a second position of the first intermediate prosody label in the second text; determining a candidate position according to the first position and the second position; determining a training data set according to the candidate position, the first audio and the first text to train a second neural network model, so that the second neural network model learns the features of the prosody label corresponding to the candidate position; and identifying the prosody label of target text using the trained second neural network model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, in particular to a prosody recognition method of text, a device and a training model method. BACKGROUND

[0002] Artificial intelligence (AI) is a comprehensive technology of computer science, which can make machines have the functions of perception, reasoning and decision-making by studying the design principles and implementation methods of various intelligent machines. With the rapid development of artificial intelligence technology, the natural language processing technology is also becoming more and more mature, and the prosody recognition function of text occupies a larger and larger proportion in the natural language processing technology.

[0003] According to the length and position of the pause, different pauses are usually represented by prosody labels #1, #2, #3 and #4, wherein #1 is the boundary of a prosodic word, representing a short pause; #2 is the boundary of a prosodic phrase, representing a lengthened sound or a short pause; #3 represents a semantically complete and more obvious pause and a tone drop; and #4 represents the end of a sentence, which is the annotation of the end of each numbered sentence.

[0004] At present, the prosody recognition scheme of text usually trains a neural network model using text annotated with prosody labels, and uses the trained neural network model to recognize the prosody of text. However, this method only recognizes the prosody of text based on the semantic features of text, and the prosody recognition result of text is not applicable in speech synthesis technology. Since the audio recorder cannot read completely according to the prosody divided according to the semantic features of text when making training data required for training a speech synthesis model, the prosody in the recorded audio file does not match the prosody of text recognized based on the semantic features of text. Training a speech synthesis model with such prosody-mismatched speech audio and annotated text makes the model unable to learn the features of prosody, so that the synthesized speech cannot clearly reflect the pauses represented by the above-mentioned prosody labels.

[0005] Therefore, how to recognize the prosody of text that matches the prosody of the recorded speech audio becomes a problem to be solved. SUMMARY

[0006] The present application provides a prosody recognition method of text, a device and a training model method, which can accurately match the prosody of the recorded audio and the prosody labels in the annotated text.

[0007] In a first aspect, a prosody recognition method of text is provided, comprising:

[0008] obtaining a first audio recorded by a first speaker according to a first text;

[0009] determine a first position of the first text according to the first audio and the first text, the first position being a position between characters or between a character and a punctuation mark in the first text corresponding to a time period of a pause of the first speaker in the first audio;

[0010] input the first text into the first neural network model to predict a first intermediate prosody label, obtain a second text, the second text comprising the first intermediate prosody label and the first text;

[0011] determine a second position of the first intermediate prosody label in the second text;

[0012] determine a candidate position according to the first position and the second position;

[0013] determine a training data set according to the candidate position, the first audio and the first text, the training data set comprising a prosody label corresponding to the candidate position;

[0014] train the second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosody label;

[0015] use the trained second neural network model to identify a prosody label of a target text.

[0016] In one example, determining the training data set according to the candidate position, the first audio and the first text comprises:

[0017] determine at least one segment of the first text and at least one segment of the first audio according to the candidate position, the at least one segment of the first text corresponding to the at least one segment of the first audio one by one, each segment of the at least one segment of the first text comprising a marker of the candidate position;

[0018] obtain a prosody label corresponding to the candidate position of each segment of the at least one segment of the first text;

[0019] determine the training data set according to the at least one segment of the first text, the prosody label corresponding to the candidate position of each segment and the at least one segment of the first audio.

[0020] In one example, the second neural network model comprises a first encoder, a second encoder, a multi-modal fusioner and a prosody label classifier, and training the second neural network model according to the training data set comprises:

[0021] use the first encoder to encode the first text segment to obtain a first text vector representation of the first text segment, the first text segment being included in the at least one segment of the first text;

[0022] encoding the first audio segment using a second encoder to obtain a first audio vector representation of the first audio segment, the first audio segment being included in at least one segment of the first audio, the first text segment corresponding to the first audio segment;

[0023] using a multi-modal fusioner to perform feature fusion on the first text vector representation and the first audio vector representation to obtain a multi-modal vector representation;

[0024] using the prosody label classifier to perform calculation on the multi-modal vector representation to obtain a predicted prosody label corresponding to the candidate position of the first text segment;

[0025] updating parameters of the second neural network model according to the predicted prosody label and a prosody label corresponding to the candidate position of the first text segment.

[0026] In one example, determining a first position of the first text according to the first audio and the first text includes:

[0027] obtaining a pause in the first audio according to the first text and the first audio based on a Montreal Forced Aligner (MFA) algorithm;

[0028] determining the first position of the first text according to the pause.

[0029] In one example, identifying a prosody label of a target text using the trained second neural network model includes:

[0030] obtaining target audio recorded by a second speaker according to the target text;

[0031] determining a third position of the target text according to the target audio and the target text, the third position being a position between characters or between a character and a punctuation mark in the target audio corresponding to a time period of a pause of the second speaker;

[0032] inputting the target text into the first neural network model to predict a second intermediate prosody label to obtain an intermediate target text, the intermediate target text being the target text including the second intermediate prosody label;

[0033] determining a fourth position of the second intermediate prosody label in the intermediate target text;

[0034] determining a candidate position of the target text according to the third position and the fourth position;

[0035] updating the target text according to the candidate position of the target text, the updated target text including a mark of the candidate position of the target text;

[0036] identifying a prosody label of the updated target text using the trained second neural network model.

[0037] In an example, the prosodic label of the updated target text is identified using the trained second neural network model, including:

[0038] According to the candidate position of the target text, at least one segment of the updated target text and at least one segment of the target audio are determined, the at least one segment of the updated target text and the at least one segment of the target audio corresponding one-to-one, each segment of the at least one segment of the updated target text including a mark of the candidate position of the target text;

[0039] The prosodic label corresponding to the candidate position of each segment of the at least one segment of the updated target text is identified using the trained second neural network model;

[0040] According to the prosodic label corresponding to the candidate position of each segment of the at least one segment of the updated target text, the prosodic label of the updated target text is determined.

[0041] In an example, the first neural network model is a bidirectional long short-term memory-conditional random field model or a span-based prosodic structure prediction model.

[0042] In an example, according to the first position and the second position, the candidate position is determined, including:

[0043] According to the first position and the second position, a position different from the first position in the second position is determined;

[0044] According to the first position and the position different from the first position in the second position, the candidate position is determined.

[0045] In a second aspect, the present application provides a prosodic recognition device of text, including:

[0046] The data acquisition module is configured to acquire a first audio recorded by a first speaker according to a first text;

[0047] The first position determination module is configured to determine a first position of the first text according to the first audio and the first text, the first position being a position between texts or between a text and a punctuation mark corresponding to a time period of a pause of the first speaker in the first audio;

[0048] The second position determination module is configured to input the first text into the first neural network model to predict a first intermediate prosodic label, and acquire a second text, the second text including the first intermediate prosodic label and the first text;

[0049] The second position determination module is further configured to determine a second position of the first intermediate prosodic label in the second text;

[0050] The candidate position determination module is configured to determine a candidate position according to the first position and the second position.

[0051] a training data set determination module configured to determine a training data set according to the candidate position, the first audio, and the first text, the training data set comprising a prosody label corresponding to the candidate position;

[0052] a neural network model training module configured to train a second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosody label;

[0053] a prosody label identification module configured to identify a prosody label of the target text using the trained second neural network model.

[0054] In a third aspect, the present application provides a method for training a model, based on the target text annotated with the prosody label obtained in any of the above embodiments, the method comprising:

[0055] obtaining target audio recorded by a second speaker according to the target text annotated with the prosody label;

[0056] obtaining a mel-frequency cepstral coefficient feature of the target audio;

[0057] determining a training data set according to the target text annotated with the prosody label and the mel-frequency cepstral coefficient feature of the target audio;

[0058] training a speech synthesis model based on the training data set, and obtaining a trained speech synthesis model.

[0059] From the above embodiments, according to the time period of the pause in the first audio, the time period of the pause is determined to correspond to a first position in the first text, and the first position is related to the pause information in the audio. Then, the first intermediate prosody label of the first text is predicted through the first neural network model, and the second position determined according to the first intermediate prosody label is related to the semantic feature. Therefore, the candidate position determined according to the first position and the second position has the pause feature in the first audio and the semantic feature in the first text, and the candidate position has the characteristics of accuracy and comprehensiveness. Further, the prosody label of the training data set determined according to the candidate position, the first audio and the first text has the characteristics of accuracy and comprehensiveness. The second neural network model trained according to the training data set can learn the features of the candidate position corresponding to the prosody label determined according to the pause information in the audio, and can also learn the features of the candidate position corresponding to the prosody label determined according to the semantic feature. Further, the trained second neural network model can combine the pause feature of the audio and the semantic feature of the text to identify the prosody label of the target text, so that the identified prosody label of the target text matches the pause in the target audio recorded according to the target text. The speech synthesis model is trained using the target audio with the matched prosody and the target text labeled with the prosody label, so that the speech synthesis model can accurately learn the features of the prosody, so that the synthesized speech can clearly reflect the pause represented by each prosody label. BRIEF DESCRIPTION OF DRAWINGS

[0060] In order to more clearly illustrate the technical solutions of the present application, the following will briefly introduce the drawings needed in the embodiments. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0061] Figure 1 is a schematic flow chart of a text prosody recognition method provided by an embodiment of the present application;

[0062] Figure 2 is a schematic diagram of a candidate position of a text provided by an embodiment of the present application;

[0063] Figure 3 is another schematic diagram of a candidate position of a text provided by an embodiment of the present application;

[0064] Figure 4 is a schematic diagram of a training sample provided by an embodiment of the present application;

[0065] Figure 5 is another schematic diagram of a training sample provided by an embodiment of the present application;

[0066] Figure 6is a schematic flowchart of a method for identifying the prosody of a text provided by an embodiment of the present application. DETAILED DESCRIPTION

[0067] Embodiments of the present application are described in detail below with reference to examples illustrated in the accompanying drawings, in which the same or similar numerals or labels represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are intended only for explanation of the present application, and cannot be interpreted as limiting the present application. It should be noted that the embodiments and features in the present application can be combined with each other unless they conflict.

[0068] Those skilled in the art can understand that, unless specifically stated, the singular forms "a", "an" and "the" used herein also include the plural forms. It should be further understood that the phrase "comprising" used in the specification of the present application means that the features, integers, steps, operations, elements and / or components exist, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we say an element is "connected" or "coupled" to another element, it can be directly connected or coupled to the other element, or there can be intermediate elements. In addition, "connected" or "coupled" used herein can include wireless connection or wireless coupling. The phrase "and / or" used herein includes all or any single unit and all combinations of the associated listed items.

[0069] Currently, prosody annotation schemes are mainly divided into manual prosody annotation according to text semantics or automatic prosody annotation according to text semantics. However, the former has high annotation cost, and the latter cannot match the prosody of the audio recorded by the speaker. Using such prosody-mismatched speech audio and prosody-annotated text to train a speech synthesis model will make the model unable to learn the characteristics of the prosody, so that the synthesized speech cannot accurately reflect the pauses represented by each prosody label.

[0070] Figure 1 is a schematic flowchart of a method for identifying the prosody of a text provided by an embodiment of the present application, in order to solve the above problems, an embodiment of the present application provides a method for identifying the prosody of a text, which will be described below in combination with Figure 1 The method is described.

[0071] S110, obtaining a first audio recorded by a first speaker according to a first text.

[0072] S120, determining a first position of the first text according to the first audio and the first text.

[0073] The first position is a position between two Chinese characters or between a Chinese character and a punctuation mark in the first text.

[0074] For example Figure 2 As shown in the figure, the first text is “Welcome to use XXXX specific person voice color speech synthesis system”, and the first audio pause time period includes t1, t2, t3, t4, t5 and t6. Among them, t1 corresponds to the position A between the fourth character “use” and the fifth character “X”, which can be recorded as “45”, t2 corresponds to the position B between the eighth character “X” and the ninth character “specific”, which can be recorded as “89”, t3 corresponds to the position C between the twelfth character “voice” and the thirteenth character “color”, which can be recorded as “1213”, t4 corresponds to the position D between the thirteenth character “color” and the fourteenth character “language”, which can be recorded as “1314”, t5 corresponds to the position E between the fifteenth character “voice” and the sixteenth character “synthesis”, which can be recorded as “1516”, and t6 corresponds to the position F after the nineteenth character “system”, which can be recorded as “19e”. “e” represents the end of the sentence. Therefore, the first position of the first text includes [(45), (89), (1213), (1314), (1516), (19e)].

[0075] In one example, the manner of determining the first position of the first text according to the first audio and the first text includes:

[0076] Based on the Montreal Forced Aligner (MFA) algorithm, the pauses in the first audio are obtained according to the first text and the first audio, and then the first position of the first text is determined according to the pauses.

[0077] Specifically, the first text and the first audio are input into the MFA. Since the MFA can analyze and cut the first audio and the first text, it can perform alignment of the first audio and the first text at the character level. In the text sequence generated by the MFA, since there is no prosodic label representing the pause in the first text, if there is a pause between two characters or between a character and a punctuation mark, it will be represented as “Xmin = …, Xmax =…, word =‘’”. By identifying the keyword “word =‘’” (‘’ represents a blank character), the position of the pause in the first text can be determined, i.e., between which two characters or between which character and which character.

[0078] In the above manner, the MFA can accurately identify the first position in the first text.

[0079] S130, input the first text into the first neural network model to predict a first intermediate prosodic label, and obtain a second text.

[0080] The second text includes the first intermediate prosodic label and the first text.

[0081] For example, the first text is "Welcome to use XXXX specific person voice color synthesis system", and the second text is "Welcome to use XXXX #2 specific person voice color synthesis system #3".

[0082] Exemplarily, the first neural network model is a bidirectional long short-term memory-conditional random field (BiLSTM-CRF) model or a span prosodic structure prediction (SpanPSP) model.

[0083] The BiLSTM-CRF model mainly includes three parts: a word embedding layer, a BiLSTM layer and a CRF layer. The word embedding layer is used to encode the input text according to the task of prosodic prediction. The encoded sequence is input into the BiLSTM layer, which can fully utilize the context information of the input sequence to extract features. The sequence output by the BiLSTM layer is input into the CRF layer to predict the category of the first intermediate prosodic label of the input text, and the first text with the first intermediate prosodic label is output, thereby obtaining the second text.

[0084] SpanPSP unifies the prediction tasks of prosodic word, prosodic phrase and intonation phrase boundaries into one by finding an optimal prosodic structure tree, directly accepts Chinese characters as input in an end-to-end manner without any preprocessing such as word segmentation and part-of-speech tagging, and then outputs the text with all levels of prosodic labels, i.e. the second text. The level of prosodic structure label can include 4 levels, represented by #1, #2, #3 and #4, and the prosodic labels #1, #2, #3 and #4 represent different levels of pauses, wherein #1 is the boundary of prosodic word, representing a short pause; #2 is the boundary of prosodic phrase, representing a lengthened tone or a short pause; #3 represents a semantically complete and more obvious pause and a tone drop; and #4 represents the end of a sentence, which is the label of the end of each numbered sentence.

[0085] In the above manner, the first intermediate prosodic label of the first text is predicted based on semantic features by the neural network model, so as to facilitate subsequent determination of the candidate position in combination with the position of the first intermediate prosodic label.

[0086] S140, determine a second position of the first intermediate prosodic label in the second text.

[0087] Taking the first text "Welcome to use XXXX specific person voice synthesis system" and the second text "Welcome to use XXXX #2 specific person voice synthesis system #3" as examples, the first intermediate prosody label is "#2" and "#3", the position of "#2" is "89", and the position of "#3" is "19e". The second position includes [(89), (19e)].

[0088] S150, determining a candidate position according to the first position and the second position.

[0089] In one example, first, the position different from the first position in the second position is determined according to the first position and the second position, and then the candidate position is determined according to the first position and the position different from the first position in the second position. Alternatively, first, the position different from the second position in the first position is determined according to the first position and the second position, and then the candidate position is determined according to the second position and the position different from the second position in the first position.

[0090] Exemplarily, the candidate position is determined according to the position different from the first position in the first position and the second position, including:

[0091] The position different from the first position in the first position and the second position is merged, and the merged position is the candidate position.

[0092] For example, taking the first position including [(45), (89), (1213), (1314), (1516), (19e)] and the second position including [(89), (19e)] as examples, there is no position different from the first position in the second position, so the merged position (i.e. the candidate position) is [(45), (89), (1213), (1314), (1516), (19e)].

[0093] In the above manner, the first position and the second position are comprehensively considered, so that the determined candidate position is more accurate.

[0094] S160, determining a training data set according to the candidate position, the first audio and the first text, the training data set including the prosody label corresponding to the candidate position.

[0095] In one example, first, at least one segment of the first text and at least one segment of the first audio are determined according to the candidate positions, the at least one segment of the first text and the at least one segment of the first audio corresponding one by one, each of the at least one segment of the first text comprising a mark of a candidate position. Then, the prosodic label corresponding to the candidate position of each of the at least one segment of the first text is obtained. Then, the training data set is determined according to the at least one segment of the first text, the prosodic label corresponding to the candidate position of each of the at least one segment of the first text, and the at least one segment of the first audio.

[0096] In one example, first, at least one segment of the first text and at least one segment of the first audio are determined according to the candidate positions, the at least one segment of the first text and the at least one segment of the first audio corresponding one by one, each of the at least one segment of the first text comprising a mark of a candidate position. Then, the prosodic label corresponding to the candidate position of each of the at least one segment of the first text is obtained. Then, the training data set is determined according to the at least one segment of the first text, the prosodic label corresponding to the candidate position of each of the at least one segment of the first text, and the at least one segment of the first audio.

[0097] Specifically, the manner of determining the at least one segment of the first text comprises: determining the context of each of the candidate positions in the first text, and determining the segment of the text corresponding to each of the candidate positions according to the context corresponding to each of the candidate positions and the mark of the position.

[0098] For example Figure 3 As shown in the figure, taking position A as an example, the context of the first text divided according to position A can be "welcome to use | XXXX specific person voice synthesis system", the context before position A is "welcome to use", and the context after position A is "XXXX specific person voice synthesis system", and the mark of position A is indicated by "|". The context of the first text divided according to position A can be "use | XXXX", the context before position A is "use", and the context after position A is "XXXX", and the mark of position A is indicated by "|". Therefore, the segment corresponding to position A comprises the segment "welcome to use | XXXX specific person voice synthesis system" and / or the segment "use | XXXX", and both of the two segments contain the mark "|", and the content of the mark is not limited in the present application. It can be seen that there can be multiple segments of the first text corresponding to the mark of each position.

[0099] Specifically, the manner of determining the at least one segment of the first audio comprises: determining the start time and the end time of each of the candidate positions in the first audio, and extracting the audio segment between the start time and the end time as the at least one segment of the first audio.

[0100] For example Figure 3 As shown in the figure, taking position A as an example, the audio segment between the end time of the word "use" and the start time of the word "X" is extracted as the segment corresponding to position A.

[0101] The way of determining the training data set is exemplified below taking position A as an example, which includes:

[0102] As shown in Figure 4 , if the segment corresponding to position A includes the segment "welcome to use | XXXX specific person voice color speech synthesis system", the segment "welcome to use | XXXX specific person voice color speech synthesis system" is labeled with the prosody label "#2", and the position of the prosody label "#2" is the position of the candidate position marker "position=4 (position starts from 0)". Among them, "sent_id" represents the serial number of the segment, "position" represents the candidate position, "left_text" represents the upper text in the segment bounded by the candidate position marker, and "right_text" represents the lower text in the segment bounded by the candidate position marker. "wav_segment" represents the name of the audio segment corresponding to the candidate position, and "label" represents the prosody label at the candidate position.

[0103] If the segment corresponding to position A includes the segment "use | XXXX", the segment "use | XXXX" is labeled with the prosody label "#2", and the position of the prosody label "#2" is the position of the candidate position marker "position=2 (position starts from 0)", as shown in Figure 5 .

[0104] If the segment corresponding to position A includes the segment "welcome to use | XXXX specific person voice color speech synthesis system" and the segment "use | XXXX", the segment "welcome to use | XXXX specific person voice color speech synthesis system" is labeled with the prosody label "#2", and the position of the prosody label "#2" is the position of the candidate position marker "position=4 (position starts from 0)". The segment "use | XXXX" is labeled with the prosody label "#2", and the position of the prosody label "#2" is the position of the candidate position marker "position=2 (position starts from 0)". Then the audio segment corresponding to the segment "welcome to use | XXXX specific person voice color speech synthesis system" is the segment corresponding to position A, and the audio segment corresponding to the segment "use | XXXX" is also the segment corresponding to position A. Both of these two groups of segments are used as data in the training data set.

[0105] In the above manner, at least one segment of the first text is determined according to the candidate position, each segment containing a marker of the candidate position, so as to facilitate subsequent learning of the second neural network model on features of the candidate position. At least one segment of the first audio is determined according to the candidate position, a start time and an end time of each segment constituting a time period corresponding to a position in the candidate position. And the segments of the first text and the segments of the first audio corresponding to each other both correspond to the position in the same candidate position, which ensures the accuracy of the training data and is conducive to the training of the candidate on the second neural network model.

[0106] S170, training the second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosodic label.

[0107] In one example, the second neural network model includes a first encoder, a second encoder, a multi-modal fusioner and a prosodic label classifier, and training the second neural network model according to the training data set includes:

[0108] encoding the first text segment using the first encoder to obtain a first text vector representation of the first text segment, the first text segment being contained in the at least one segment of the first text;

[0109] encoding the first audio segment using the second encoder to obtain a first audio vector representation of the first audio segment, the first audio segment being contained in the at least one segment of the first audio, and the first text segment corresponding to the first audio segment;

[0110] using the multi-modal fusioner to perform feature fusion on the first text vector representation and the first audio vector representation to obtain a multi-modal vector representation;

[0111] using the prosodic label classifier to calculate the multi-modal vector representation to obtain a predicted prosodic label corresponding to the candidate position of the first text segment;

[0112] updating parameters of the second neural network model according to the predicted prosodic label and the prosodic label corresponding to the candidate position of the first text segment.

[0113] Exemplarily, the first encoder adopts a Bidirectional Encoder Representation from Transformers (Bert) model of Transformers, respectively encodes the context and the context of the segment of the first text divided according to the marker of the candidate position, and then splices the encoding results of the context and the context to obtain the final encoding result of the segment. For example, Figure 5As shown, the encoding result of the preceding text is the first token (i.e., [CLS]) of the sequence output by the last layer of the Transformer in BERT. The [CLS] symbol is inserted in front of the text by the BERT model, and the output sequence corresponding to the symbol is taken as the semantic representation of the entire text, that is, the representation of the first word of each sentence is taken as the representation of the entire sentence, which is used for text classification. The encoding result of the following text is obtained in the same way as the encoding result of the preceding text.

[0114] Exemplarily, the second encoder adopts wav2vec, which is an unsupervised model for generating speech representation.

[0115] For example Figure 6 As shown, the first text segment is x a , the first audio segment is x l , the first encoder f a encodes x a to generate a vector representation z a , and the second encoder f l encodes x l to generate a vector representation z l . The multi-modal fusioner includes a first low-rank factor and a second low-rank factor , where r represents or the rank. As shown in formula (1), the steps of obtaining the multi-modal vector representation h include: the first low-rank factor and the vector representation z a to which 1 is added are subjected to dot product operation to obtain a first dot product operation result, so as to extract the features of the vector representation z a . The second low-rank factor and the vector representation z l to which 1 is added are subjected to dot product operation to obtain a second dot product operation result, so as to extract the features of the vector representation z l . Then, the first dot product operation result and the second dot product operation result are subjected to outer product operation to obtain h.

[0116] Formula (1)

[0117] wherein, denotes tensor product, denotes dot product, denotes outer product, is a tensor, and represents and after outer product operation, representing the comprehensive features of and .

[0118] Then the multi-modal vector representation h is input into a prosody label classifier to obtain a prosody label corresponding to the candidate position of the first text segment.

[0119] In the above manner, the first text segment is encoded using the first encoder to extract semantic features of the first text segment. The first audio segment is encoded using the second encoder to extract speech features of the first audio segment. The multi-modal fusioner can fuse the semantic features and the speech features, and comprehensively use information from the two modalities to make prosody label prediction. In the prediction process, a single modality usually cannot contain all the effective information required to produce accurate prediction results. The multi-modal fusion process combines information from the two modalities to supplement information, broaden the coverage of information contained in the input data, improve the accuracy of the prosody label classification result, and improve the robustness of the prosody label classifier.

[0120] S180, identifying a prosody label of the target text using the trained second neural network model.

[0121] In one example, the step of identifying a prosody label of the target text using the trained second neural network model includes:

[0122] S181, obtaining target audio recorded by the second speaker according to the target text.

[0123] S182, determining a third position of the target text according to the target audio and the target text, the third position being a position between characters or between a character and a punctuation mark in the target audio corresponding to a pause of the second speaker.

[0124] S184, inputting the target text into the first neural network model to predict a second intermediate prosody label, and obtaining an intermediate target text, the intermediate target text being the target text including the second intermediate prosody label.

[0125] S185, determining a fourth position of the second intermediate prosody label in the intermediate target text.

[0126] S186, determining a candidate position of the target text according to the third position and the fourth position.

[0127] S187, updating the target text according to the candidate position of the target text, the updated target text including a mark of the candidate position of the target text.

[0128] S188, identifying a prosody label of the updated target text using the trained second neural network model.

[0129] In the above manner, first, the third position of the pause in the target text is determined by using the time period of the pause in the target audio. Then, the prosody label of the target text is predicted based on the semantic feature by using the first neural network model, and the fourth position of the prosody label in the target text is determined. The candidate position of the target text is determined according to the third position and the fourth position. The candidate position includes both the position determined according to the pause in the audio and the position of the prosody label determined based on the semantic feature, so the candidate position is both accurate and comprehensive, and then the prosody label identified by the second neural network according to the candidate position is both accurate and comprehensive.

[0130] Further exemplarily, the prosody label of the updated target text is identified by using the trained second neural network model, including:

[0131] According to the candidate position of the target text, at least one segment of the updated target text and at least one segment of the target audio are determined, and the at least one segment of the updated target text corresponds to the at least one segment of the target audio one by one, and each segment of the at least one segment of the updated target text includes a mark of the candidate position of the target text;

[0132] The prosody label corresponding to the candidate position of each segment of the at least one segment of the updated target text is identified by using the trained second neural network model;

[0133] According to the prosody label corresponding to the candidate position of each segment of the at least one segment of the updated target text, the prosody label of the updated target text is determined.

[0134] In the above manner, according to the candidate position of the target text, the segment of the target text is determined, and the segment of the target text contains a mark of the candidate position. Then, the time period corresponding to the candidate position in the target audio is determined according to the candidate position of the target text, and the segment of the target audio is extracted. The segment of the target text and the segment of the target audio corresponding to the same candidate position are input into the trained second neural network model to obtain the prosody label corresponding to the candidate position.

[0135] According to the embodiments, the time period of the pause in the first audio is determined, and the first position corresponding to the time period of the pause in the first text is determined. The first position is related to the pause information in the audio. Then, the first intermediate prosody label of the first text is predicted by the first neural network model, and the second position determined according to the first intermediate prosody label is related to the semantic feature. Therefore, the candidate position determined according to the first position and the second position has the pause feature in the first audio and the semantic feature in the first text, and the candidate position has the characteristics of accuracy and comprehensiveness. Further, the prosody label of the training data set determined according to the candidate position, the first audio, and the first text has the characteristics of accuracy and comprehensiveness. The second neural network model trained according to the training data set can learn the features of the candidate position corresponding to the prosody label determined according to the pause information in the audio, and can also learn the features of the candidate position corresponding to the prosody label determined according to the semantic feature. Further, the trained second neural network model can identify the prosody label of the target text by combining the pause feature of the audio and the semantic feature of the text, so that the identified prosody label of the target text matches the pause in the target audio recorded according to the target text. The speech synthesis model is trained using the target audio with the matched prosody and the target text labeled with the prosody label, so that the speech synthesis model can accurately learn the features of the prosody, so that the synthesized speech can clearly reflect the pause represented by each prosody label.

[0136] In combination with the embodiments of the text prosody identification method, the application further provides a text prosody identification device, comprising:

[0137] The data acquisition module is configured to acquire the first audio recorded by the first speaker according to the first text.

[0138] The position determination module is configured to determine the first position of the first text according to the first audio and the first text. The first position is the position between the texts or between the text and the punctuation corresponding to the time period of the pause of the first speaker in the first audio.

[0139] The position determination module is further configured to input the first text into the first neural network model to predict the first intermediate prosody label, and acquire the second text. The second text comprises the first intermediate prosody label and the first text.

[0140] The position determination module is further configured to determine the second position of the first intermediate prosody label in the second text.

[0141] The candidate position determination module is configured to determine the candidate position according to the first position and the second position.

[0142] The training data set determination module is configured to determine a training data set according to the candidate position, the first audio, and the first text, the training data set comprising a prosody label corresponding to the candidate position.

[0143] The neural network model training module is configured to train the second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosody label.

[0144] The prosody label recognition module is configured to recognize the prosody label of the target text by using the trained second neural network model.

[0145] In one example, the training data set determination module comprises a text segment and audio segment acquisition sub-module, a prosody label acquisition sub-module, and a training data set determination sub-module.

[0146] The text segment and audio segment acquisition sub-module is configured to determine at least one segment of the first text and at least one segment of the first audio according to the candidate position, the at least one segment of the first text corresponding to the at least one segment of the first audio one by one, each segment of the at least one segment of the first text comprising a marker of the candidate position.

[0147] The prosody label acquisition sub-module is configured to acquire a prosody label corresponding to the candidate position of each segment of the at least one segment of the first text.

[0148] The training data set determination sub-module is configured to determine the training data set according to the at least one segment of the first text, the prosody label corresponding to the candidate position of each segment, and the at least one segment of the first audio.

[0149] In one example, the neural network model training module comprises a second neural network model.

[0150] The second neural network model comprises a first encoder, a second encoder, a multi-modal fusioner, and a prosody label classifier.

[0151] The first encoder is configured to encode the first text segment to obtain a first text vector representation of the first text segment, the first text segment being included in the at least one segment of the first text.

[0152] The second encoder is configured to encode the first audio segment to obtain a first audio vector representation of the first audio segment, the first audio segment being included in the at least one segment of the first audio, and the first text segment corresponding to the first audio segment.

[0153] The multi-modal fusioner is configured to perform feature fusion on the first text vector representation and the first audio vector representation to obtain a multi-modal vector representation.

[0154] The prosody label classifier is used to calculate a multi-modal vector representation to obtain a predicted prosody label corresponding to the candidate position of the first text segment.

[0155] The neural network model training module further includes a parameter updating module configured to update parameters of the second neural network model based on the predicted prosody label and the prosody label corresponding to the candidate position of the first text segment.

[0156] In one example, the prosody recognition device further includes a prosody label updating module, wherein,

[0157] The data acquisition module is further configured to acquire target audio recorded by the second speaker based on the target text;

[0158] The position determination module is further configured to determine a third position of the target text based on the target audio and the target text, the third position being a position between characters or between a character and a punctuation mark in the target audio corresponding to a pause of the second speaker;

[0159] The position determination module is further configured to input the target text into the first neural network model to predict a second intermediate prosody label, and obtain an intermediate target text, the intermediate target text being the target text including the second intermediate prosody label;

[0160] The position determination module is further configured to determine a fourth position of the second intermediate prosody label in the intermediate target text;

[0161] The candidate position determination module is further configured to determine a candidate position of the target text based on the third position and the fourth position;

[0162] The position determination module is further configured to update the target text based on the candidate position of the target text, the updated target text including a mark of the candidate position of the target text;

[0163] The prosody label updating module is configured to identify a prosody label of the updated target text using the trained second neural network model.

[0164] Other implementations and effects of the device are described above in the method for recognizing prosody of a text, and will not be repeated here.

[0165] In combination with the above embodiments of the method for recognizing prosody of a text, the present application further provides a method for training a model, based on the target text annotated with prosody labels obtained by the above method, the method comprising:

[0166] Acquiring target audio recorded by the second speaker based on the target text annotated with prosody labels;

[0167] Acquiring a mel-frequency cepstral coefficient feature of the target audio;

[0168] Determine a training data set according to a mel-frequency cepstral coefficient feature of the target text and the target audio annotated by the prosody label;

[0169] Train a speech synthesis model based on the training data set, and obtain the trained speech synthesis model.

[0170] The above describes the basic principles of the present application in combination with specific embodiments, but it should be noted that the advantages, advantages, effects and the like mentioned in the present application are only examples and are not limiting, and these advantages, advantages, effects and the like cannot be considered as the must-have of each embodiment of the present application. In addition, the above specific details are only for the purpose of example and for the purpose of understanding, and the above details do not limit the present application to the above specific details.

[0171] It should be understood that although each step in the flowchart of the accompanying drawings is displayed in sequence according to the direction of the arrow, these steps are not necessarily executed in sequence according to the direction of the arrow. Unless otherwise specified herein, the execution of these steps is not strictly limited in sequence, and they can be executed in other order. Moreover, at least part of the steps in the flowchart of the accompanying drawings can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times, and the execution order is not necessarily sequential, but can be executed alternately or alternately with at least part of other steps or other steps. Sub-steps or stages.

[0172] The block diagrams of the devices, apparatuses, equipment, systems involved in the present application are only illustrative examples and are not intended to require or imply the connection, arrangement, configuration shown in the block diagram. As those skilled in the art will recognize, these devices, apparatuses, equipment, systems can be connected, arranged, configured in any manner. Words such as "include", "contain", "have" and the like are open-ended words, which mean "include but not limited to", and can be used interchangeably. The words "or" and "and" used herein mean the word "and / or", and can be used interchangeably unless the context clearly indicates otherwise. The word "such as" used herein means the phrase "such as but not limited to", and can be used interchangeably.

[0173] It should also be noted that in the devices, equipment and methods of the present application, each component or each step can be decomposed and / or recombined. These decompositions and / or recombination should be considered as equivalent solutions of the present application.

[0174] The above description of the disclosed aspects is provided to enable any person skilled in the art to make or use the application. Various modifications to these aspects will be readily apparent to those skilled in the art, and the generic principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

[0175] The above description has been presented to enable any person skilled in the art to make or use the application. Numerous modifications to the aspects described herein will be readily apparent to those skilled in the art, and the principles defined herein can be applied to other aspects without departing from the scope of the application. Thus, the present application is not intended to be limited to the aspects shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of prosody recognition of text, characterized by, The method comprises: obtaining first audio recorded by a first speaker according to first text; determining a first position of the first text according to the first audio and the first text, the first position being a position between characters or between a character and a punctuation mark in the first audio corresponding to a time period in which the first speaker pauses; inputting the first text into a first neural network model to predict a first intermediate prosody label, and obtaining second text, the second text comprising the first intermediate prosody label and the first text; determining a second position of the first intermediate prosody label in the second text; determining a candidate position according to the first position and the second position, wherein the determining of the candidate position according to the first position and the second position comprises: determining a position different from the first position in the second position according to the first position and the second position; merging the first position and the position different from the first position in the second position, the merged position being the candidate position; determining a training data set according to the candidate position, the first audio and the first text, the training data set comprising a prosody label corresponding to the candidate position, wherein the determining of the training data set according to the candidate position, the first audio and the first text comprises: determining at least one segment of the first text and at least one segment of the first audio corresponding to the candidate position according to the candidate position, each segment of the at least one segment of the first text comprising a mark of the candidate position; obtaining a prosody label corresponding to the candidate position of each segment of the at least one segment of the first text; determining a training data set according to the at least one segment of the first text, the prosody label corresponding to the candidate position of each segment and the at least one segment of the first audio; training a second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosody label; identifying the prosody label of target text using the trained second neural network model.

2. The method of claim 1, wherein, The second neural network model comprises a first encoder, a second encoder, a multi-modal fusioner and a prosody label classifier, and the training of the second neural network model according to the training data set comprises: encoding a first text segment using the first encoder to obtain a first text vector representation of the first text segment, the first text segment being included in the at least one segment of the first text; encoding a first audio segment using the second encoder to obtain a first audio vector representation of the first audio segment, the first audio segment being included in the at least one segment of the first audio, the first text segment corresponding to the first audio segment; using the multi-modal fusioner to perform feature fusion on the first text vector representation and the first audio vector representation to obtain a multi-modal vector representation; performing calculation on the multi-modal vector representation using the prosody tag classifier to obtain a predicted prosody tag corresponding to a candidate position of the first text segment; updating parameters of the second neural network model according to the predicted prosody tag and a prosody tag corresponding to the candidate position of the first text segment.

3. The method according to claim 1 or 2, characterized in that, The determining the first position of the first text according to the first audio and the first text comprises: obtaining a pause in the first audio according to the first text and the first audio based on a Montreal Forced Aligner (MFA) algorithm; determining the first position of the first text according to the pause.

4. The method according to claim 1 or 2, characterized in that, The using the trained second neural network model to identify the prosody tag of the target text comprises: obtaining target audio recorded by a second speaker according to the target text; determining a third position of the target text according to the target audio and the target text, the third position being a position between texts or between a text and a punctuation symbol corresponding to a time period of a pause of the second speaker in the target audio; inputting the target text into the first neural network model to predict a second intermediate prosody tag to obtain an intermediate target text, the intermediate target text being the target text including the second intermediate prosody tag; determining a fourth position of the second intermediate prosody tag in the intermediate target text; determining a candidate position of the target text according to the third position and the fourth position; updating the target text according to the candidate position of the target text, the updated target text including a mark of the candidate position of the target text; using the trained second neural network model to identify the prosody tag of the updated target text.

5. The method of claim 4, wherein, The using the trained second neural network model to identify the prosody tag of the updated target text comprises: determining at least one segment of the updated target text and at least one segment of the target audio according to the candidate position of the target text, the at least one segment of the updated target text corresponding to the at least one segment of the target audio one by one, each of the at least one segment of the updated target text including the mark of the candidate position of the target text; using the trained second neural network model to identify the prosody tag corresponding to the candidate position of each of the at least one segment of the updated target text; determining the prosody tag of the updated target text according to the prosody tag corresponding to the candidate position of each of the at least one segment of the updated target text.

6. The method of claim 1 or 2, wherein, The first neural network model is a bidirectional long short-term memory-conditional random field model or a span-based prosody structure prediction model.

7. A prosody recognition apparatus of a text, characterized by comprising: comprises: a data acquisition module configured to acquire first audio recorded by a first speaker according to first text; The first position determining module is configured to determine a first position of the first text according to the first audio and the first text, the first position being a position between characters or between a character and a punctuation mark corresponding to a time period of a pause of the first speaker in the first audio. The second position determining module is configured to input the first text into a first neural network model to predict a first intermediate prosody label, and obtain a second text, the second text comprising the first intermediate prosody label and the first text. The second position determining module is further configured to determine a second position of the first intermediate prosody label in the second text. The candidate position determining module is configured to determine a candidate position according to the first position and the second position, wherein the determination of the candidate position according to the first position and the second position comprises: determining a position different from the first position in the second position according to the first position and the second position; merging the first position and the position different from the first position in the second position, the merged position being the candidate position. The training data set determining module is configured to determine a training data set according to the candidate position, the first audio, and the first text, the training data set comprising a prosody label corresponding to the candidate position, wherein the determination of the training data set according to the candidate position, the first audio, and the first text comprises: determining at least one segment of the first text and at least one segment of the first audio according to the candidate position, each segment of the at least one segment of the first text corresponding to one of the at least one segment of the first audio, each segment of the at least one segment of the first text comprising a mark of the candidate position; obtaining a prosody label corresponding to the candidate position of each segment of the at least one segment of the first text; determining the training data set according to the at least one segment of the first text, the prosody label corresponding to the candidate position of each segment, and the at least one segment of the first audio. The neural network model training module is configured to train a second neural network model according to the training data set, so that the second neural network model learns features of the candidate position corresponding to the prosody label. The prosody label identifying module is configured to identify the prosody label of a target text using the trained second neural network model.

8. A method of training a model, the method comprising: The method comprises: obtaining a first audio recorded by a first speaker according to a first text; determining a first position of the first text according to the first audio and the first text, the first position being a position between characters or between a character and a punctuation mark corresponding to a time period of a pause of the first speaker in the first audio. inputting the first text into a first neural network model to predict a first intermediate prosody label, and obtaining a second text, the second text comprising the first intermediate prosody label and the first text. determining a second position of the first intermediate prosody label in the second text. According to the first position and the second position, a candidate position is determined, wherein the determining of the candidate position according to the first position and the second position comprises: According to the first position and the second position, a position different from the first position in the second position is determined; The first position and the position different from the first position in the second position are merged, and the merged position is the candidate position; According to the candidate position, the first audio and the first text, a training data set is determined, and the training data set comprises a prosody label corresponding to the candidate position, wherein the determining of the training data set according to the candidate position, the first audio and the first text comprises: According to the candidate position, at least one segment of the first text and at least one segment of the first audio are determined, the at least one segment of the first text and the at least one segment of the first audio correspond one by one, and each segment of the at least one segment of the first text comprises a mark of the candidate position; The prosody label corresponding to the candidate position in each segment of the at least one segment of the first text is obtained; According to the at least one segment of the first text, the prosody label corresponding to the candidate position of each segment and the at least one segment of the first audio, a training data set is determined; According to the training data set, a second neural network model is trained, so that the second neural network model learns the features of the candidate position corresponding to the prosody label; The prosody label of the target text is identified by using the trained second neural network model; The target audio recorded by a second speaker according to the target text labeled by the prosody label is obtained; The mel-frequency cepstral coefficient feature of the target audio is obtained; According to the target text labeled by the prosody label and the mel-frequency cepstral coefficient feature of the target audio, a training data set is determined; Based on the training data set, a speech synthesis model is trained, and a trained speech synthesis model is obtained.

Images