Punctuation and capitalization of speech recognition transcripts

Abstract

1. A method comprising: receiving a first text corpus comprising punctuated and capitalized text; annotating words in the first text corpus with a set of labels indicating the punctuation and capitalization of each word; during an initial training phase, training a machine learning model on a first training set comprising (i) the annotated words in the first text corpus and (ii) the labels; receiving a second text corpus representing conversational speech; annotating words in the second text corpus with the set of labels; during a retraining phase, retraining the machine learning model on a second training set comprising (iii) the annotated words in the second text corpus and (iv) the labels; and during an inference phase, applying the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

Description

Technical Field

[0001] (Cross - reference to related applications and claim of priority) This application claims priority to U.S. Patent Application No. 17 / 135,283, titled "PUNCTUATION AND CAPITALIZATION OF SPEECH RECOGNITION TRANSCRIPTS", filed on December 28, 2020.

Background Art

[0002] In call center analysis, speech recognition is used as the first step in the analysis of these conversions, for example, to detect important call events, customer sentiment, or to summarize the content of a conversation, by transcribing the conversation between an agent and a customer. Another common use case for automatic transcription of telephone conversations in a call center is, for example, to perform quality control of the telephone conversation content by a supervisor.

[0003] [[ID==18]]Conventionally, speech recognition results do not include punctuation and capitalization of text. As a result, automatically generated transcripts are more difficult to read than human - generated transcripts that are more frequently punctuated and capitalized.

[0004] When the recognized text is further processed by downstream natural language processing (NLP) applications, in addition to being easier to read, punctuation and capitalization are important. For example, a named - entity recognition device clearly benefits from the capitalization of names and locations that makes it easier to recognize those entities.

[0005] The foregoing examples of related art and the limitations related thereto are intended to be illustrative and not exclusive. Other limitations of the related art will become apparent to those of ordinary skill in the art upon reading this specification and examining the drawings.

Summary of the Invention

[0006] The following embodiments and aspects thereof are intended to be illustrative and illustrative, and are described and illustrated in conjunction with other systems, tools, and methods, without limiting their scope.

[0007] In one embodiment, a system comprising at least one hardware processor and a non-temporary computer-readable storage medium storing program instructions, wherein the program instructions include: receiving a first text corpus containing punctuated and capitalized text by at least one hardware processor; annotating words in the first text corpus with a set of labels, the labels indicating the punctuation and capitalization associated with each word in the first text corpus; and training a machine learning model on a first training set in an initial training phase, the first training set including (i) annotated words in the first text corpus and (ii) labels. A system is provided, which is a program instruction executable to: receive a second text corpus representing conversational speech; annotate words in the second text corpus with a set of labels, the labels indicating punctuation and capitalization associated with each word in the second text corpus; retrain a machine learning model on a second training set, the second training set comprising (iii) annotated words in the second text corpus and (iv) labels; and in the inference stage, apply the trained machine learning model to a target set of words representing conversational speech to predict punctuation and capitalization for each word in the target set.

[0008] In one embodiment, a method comprising: receiving a first text corpus containing punctuation and capitalized text; annotating words in the first text corpus with a set of labels, the labels indicating punctuation and capitalization associated with each word in the first text corpus; training a machine learning model on a first training set in an initial training phase, the first training set comprising (i) annotated words in the first text corpus and (ii) labels; receiving a second text corpus representing conversational speech; and training the second text corpus A method is also provided which includes annotating words in a path with a set of labels, the labels indicating punctuation and capitalization associated with each word in a second text corpus; retraining a machine learning model on a second training set, the second training set comprising (iii) annotated words in the second text corpus and (iv) labels; and in an inference stage, applying the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

[0009] In one embodiment, a computer program product comprising a non-temporary computer-readable storage medium on which program instructions are embodied, wherein the program instructions include: receiving a first text corpus containing punctuated and capitalized text by at least one hardware processor; annotating words in the first text corpus with a set of labels, the labels indicating punctuation and capitalization associated with each word in the first text corpus; and training a machine learning model on a first training set in an initial training stage, the first training set including (i) annotated words in the first text corpus and (ii) labels; and training conversational speech Further provided is a computer program product that can perform the following: receiving a second text corpus representing; annotating the words in the second text corpus with a set of labels, the labels indicating punctuation and capitalization associated with each word in the second text corpus; retraining a machine learning model on a second training set, the second training set comprising (iii) annotated words in the second text corpus and (iv) labels; and in an inference stage, applying the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

[0010] In some embodiments, the label indicating punctuation is selected from the group consisting of commas, periods, question marks, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

[0011] In some embodiments, the first text corpus is preprocessed before training by converting at least all words in the first text corpus to lowercase.

[0012] In some embodiments, the second text corpus is preprocessed by performing contextualization before retraining, which includes segmenting the text corpus into segments, each containing at least two sentences.

[0013] In some embodiments, a second text corpus is preprocessed before retraining by performing data augmentation, which includes extending at least some segments by adding at least one preceding sentence in the conversational audio and at least one following sentence in the conversational audio.

[0014] In some embodiments, the prediction includes confidence scores associated with each of the predicted punctuation and predicted capitalization, and when a word in the target set is included in two or more segments and two or more predictions regarding punctuation or capitalization are received, the confidence scores associated with the two or more predictions are averaged to produce a final confidence score for the prediction.

[0015] In some embodiments, the second text corpus is preprocessed before retraining by including end-of-sentence (EOS) embeddings.

[0016] In some embodiments, the second text corpus and word target set each include transcribed text representing a conversation between at least two participants, the at least two participants being a call center agent and a customer.

[0017] In some embodiments, transcription includes at least one analysis selected from the group consisting of text detection, speech recognition, and speech-to-text detection.

[0018] In one embodiment, a system is provided comprising at least one hardware processor and a non-temporary computer-readable storage medium storing program instructions, wherein the program instructions are executable by the at least one hardware processor to perform operations on a multitask neural network, the multitask neural network comprising: a capitalization prediction network that takes a text corpus containing at least one sentence as input and predicts the capitalization of each word in at least one sentence, and is trained on a first loss function; a punctuation prediction network that takes a text corpus as input and predicts punctuation on the text corpus, and is trained on a second loss function; and an output layer that outputs coordinated predictions of capitalization and punctuation based on a multitask loss function that combines the first and second loss functions, wherein the capitalization prediction network and the punctuation prediction network are trained in cooperation.

[0019] In some embodiments, the program instructions can be further executed during the inference phase to apply a multitasking neural network to a target set of words representing conversational speech to predict punctuation and capitalization of each word in the target set.

[0020] In some embodiments, the coordinated training includes, in an initial training phase, training a capitalization prediction network and a punctuation prediction network in conjunction with a first training set, the first training set comprising (i) a first text corpus containing punctuated and capitalized text, and (ii) labels indicating punctuation and capitalization associated with each word in the first text corpus.

[0021] In some embodiments, the collaborative training further includes, in the retraining phase, training the capitalization prediction network and the punctuation prediction network in collaboration with respect to a second training set, the second training set including (i) a second text corpus representing conversational speech and (ii) labels indicating punctuation and capitalization associated with each of the words in the second text corpus.

[0022] In some embodiments, the label indicating punctuation is selected from the group consisting of comma, period, question mark, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

[0023] In some embodiments, the first text corpus is preprocessed by converting at least all the words in the first text corpus to lowercase before training.

[0024] In some embodiments, the second text corpus is preprocessed by performing contextualization before retraining, the contextualization including segmenting the text corpus into segments each including at least two sentences.

[0025] In some embodiments, the second text corpus is preprocessed by performing data augmentation before retraining, the data augmentation including extending at least some of the segments by adding at least one of one or more preceding sentences and one or more subsequent sentences in the conversational speech.

[0026] In some embodiments, the second text corpus is preprocessed by including end-of-sentence (EOS) embedding before retraining.

[0027] In addition to the exemplary aspects and embodiments described above, further aspects and embodiments will become apparent by referring to the drawings and examining the following detailed description.

Brief Description of the Drawings

[0028] Exemplary embodiments are illustrated in the figures referred to. The dimensions of the components and features shown in the figures are generally selected for the convenience and clarity of presentation and are not necessarily shown to scale. The drawings are listed below. [Figure 1] Schematically illustrates a model for jointly predicting punctuation and capitalization according to some embodiments. [Figure 2A] A flowchart of functional steps of a process of the present disclosure for training to generate a machine learning model for automatic prediction of punctuation and capitalization in a transcribed text according to some embodiments. [Figure 2B] A schematic illustration of data processing steps combined with constructing one or more machine learning training datasets of the present disclosure according to some embodiments. [Figure 3] A schematic illustration of a neural network structure including end-of-sentence embedding that can be employed in the context of a machine learning model of the present disclosure according to some embodiments. [Figure 4] A schematic illustration of a neural network structure for jointly predicting punctuation and capitalization according to some embodiments.

MODE FOR CARRYING OUT THE INVENTION

[0029] This specification discloses a method, a system, and a computer program product for automatic prediction of punctuation and capitalization in a transcribed text. In some embodiments, the present disclosure is particularly suitable for automatic punctuation and capitalization of conversational voice transcriptions, particularly in the context of automatic transcription of contact center conversations, for example.

[0030] Automatic speech recognition (ASR) systems are being widely adopted in a variety of applications, including voice commands, voice assistants, dictation tools, and conversational transcripters. A significant limitation in many ASRs is the lack of punctuation or capitalization in the transcribed text. This can be problematic when the output is presented visually, as unpunctuated transcripts are more difficult to read and understand, and when these transcripts are used as input for downstream tasks, such as those in the field of natural language processing (NLP). For example, typical NLP systems are usually trained on punctuated text, and therefore, the lack of punctuation can lead to a significant degradation in the system's performance.

[0031] Typically, punctuation and capitalization tasks are solved using supervised machine learning methods. Such models may use a transcribed and punctuated speech corpus to train a machine learning model to predict text punctuation using a set of features, e.g., the text itself, speaker input instructions, and / or timing input. Other approaches may rely on an inter-sequence network architecture, in which case the input is a sequence of lowercase, unpunctuated words and the output is a sequence with corrected capitalization and punctuation inserted.

[0032] In some embodiments, the disclosure provides the ability to add punctuation and capitalization to an automated transcript, which may be particularly suitable for use in conjunction with a transcript of a multi-turn call center conversation, for example, representing a back-and-forth dialogue between a customer and an agent.

[0033] In some embodiments, the Disclosure provides a supervised machine learning model trained using a two-step training process, wherein (i) the first step uses a large amount of punctuated and capitalized text from a provided corpus, e.g., from readily available and economical sources such as internet text, and (ii) the second step uses a relatively small amount of dialogue transcript annotated for punctuation and capitalization, which is more costly to acquire due to the cost of manual annotation. In some embodiments, the second training step employs a material augmentation mechanism that provides contextual information about the text in the training dataset. In some embodiments, the material augmentation may also employ sentence-end embedding.

[0034] In some embodiments, this machine learning model is based on a unique neural network architecture configured for multi-task training. Multi-task learning, or training, is a category of machine learning tasks in which multiple learning tasks are solved simultaneously while leveraging commonalities across tasks. This can improve the learning efficiency and predictive accuracy of task-specific models compared to training models separately. A multi-task machine learning model learns two or more tasks in parallel using shared representations, and what is learned for each task can help the other tasks be learned better. In a classification context, multi-task learning aims to improve the performance of multiple classification tasks by learning them in conjunction.

[0035] Accordingly, in some embodiments, the Disclosure provides a machine learning model that uses a neural network architecture configured to learn capitalization and punctuation in conjunction, where the conjunction learning provides a potential information gain over separate capitalization and punctuation models. In some embodiments, such a machine learning model leverages a strong interdependence between two learning tasks. For example, capitalized words often follow periods, and punctuation information such as question marks or periods may indicate that the following word should be capitalized.

[0036] In some embodiments, the Disclosure provides training a collaborative model using a training corpus that includes (i) punctuated and capitalized general text and (ii) intradomain multi-turn dialogue annotated for punctuation and capitalization. In some embodiments, the collaborative machine learning model performs several separate machine learning tasks and comprises a capitalization machine learning classifier that predicts capitalized labels for target words or tokens and a punctuation machine learning model that predicts punctuation labels.

[0037] As schematically illustrated in Figure 1, in some embodiments, the Disclosure provides a single machine learning model for coordinating prediction of punctuation and capitalization, where the model's loss function optimally weights each task. By using a single model, the Disclosure provides more consistent output and improved accuracy, for example, when capitalization may depend on the results of nearby punctuation predictions. In addition, combining both tasks into a single model may provide reduced computational overhead and better model performance.

[0038] In some embodiments, the disclosure uses sequence tagging, which is defined as a type of pattern recognition task that includes the automatic assignment of class labels to each member of a sequence of observations.

[0039] In the context of speech recognition, sequence tagging can include part-of-speech (POS) tagging, which is the process of marking words in text as corresponding to a specific part of speech based on both their definition and their context, such as identifying them as nouns, verbs, adjectives, adverbs, etc. Sequence tagging can also include other NLP tasks such as chunking and named entity recognition (NER).

[0040] Most sequence labeling algorithms are inherently probabilistic and rely on statistical reasoning to find the best sequence. The most common statistical models used in sequence labeling make the Markov assumption, that is, the assumption that the choice of label for a given word depends only directly on the labels that are directly adjacent to it. Thus, the set of labels forms a Markov chain. This naturally leads to the hidden Markov model (HMM), one of the most common statistical models used in sequence labeling. Other common models used are the maximum entropy Markov model and the conditional random field.

[0041] In some embodiments, the Disclosure provides one or more neural network-based machine learning models trained to perform a sequence tagging task. In some embodiments, these models may include one or more Long Short-Term Memory (LSTM) networks, bidirectional LSTM networks (BiLSTM), LSTM networks with a CRF layer (LSTM-CRF), and / or bidirectional LSTM networks with a Conditional Random Field (CRF) layer (BILSTM-CRF).

[0042] In some embodiments, the trained machine learning model of the Disclosure may be configured to take a sequence of words as input and, for each word in the sequence, output a predicted punctuation tag from a set of punctuation tags, the punctuation tag indicating a punctuation action to be performed on the word, for example:

[0043] [Table 1]

[0044] In some embodiments, the trained machine learning model of the Disclosure may be configured to take a sequence of words as input and, for each word in the sequence, output a predicted capitalization tag for that word from a closed set of capitalization tags, where the capitalization tag indicates a capitalization action to be performed on the word, for example:

[0045] [Table 2]

[0046] Figure 2A is a flowchart of the functional steps of the process of this disclosure for training to generate a machine learning model for automatic prediction of punctuation and capitalization in transcribed text, according to several embodiments.

[0047] Figure 2B is a schematic example of data processing steps, including constructing one or more machine learning training datasets according to several embodiments of this disclosure.

[0048] In some embodiments, in step 200, the first training dataset of the present disclosure may be generated using, for example, a corpus of general text provided from available proprietary resources and / or public sources. In some embodiments, the provided text is punctuated and capitalized text. In some embodiments, the provided text is annotated with corresponding punctuation and capitalization annotations, which may be performed manually by an annotation specialist.

[0049] In some embodiments, the provided corpus undergoes selection and / or filtering to extract a subset of text, for example, by filtering based on language and / or other criteria. In some embodiments, this step removes noise and irrelevant material, which helps to make training faster and less susceptible to the negative effects of noise.

[0050] In some embodiments, the Disclosure uses a language modeling approach that employs a speech recognition language model to select a relevant subset from a provided corpus, and the model predicts the probability that an input sentence is the result of a speech recognition process applied to domain-specific (e.g., call center) speech. In some embodiments, the Disclosure may employ a word count model, where for each sentence in the provided corpus, the model counts how many of the words in the sentence match entries in a known dictionary (e.g., a domain-specific strain containing typical call center vocabulary), and may select only sentences that contain lexical words above a specified threshold (e.g., 80%).

[0051] In some embodiments, in step 202, the provided text corpus may be preprocessed, for example, to normalize and / or standardize the text within the corpus. For example, preprocessing may be applied to convert all words to lowercase and / or tag each word with corresponding punctuation and capitalization tags. For example, in some embodiments, the sentence "Hi, how can I help you?" may be transformed as follows:

[0052] [Table 3]

[0053] In some embodiments, the preprocessing step of the present disclosure may generate a corpus of sentences in which all entities (words) within the corpus are presented uniformly (e.g., in lowercase).

[0054] In some embodiments, in step 204, the first training dataset may be used to perform preliminary training of the machine learning model of the Disclosure. In some embodiments, for example, the pre-trained machine learning model of the Disclosure, trained on the first training dataset, may be configured to predict punctuation and capitalization in transcribed text, for example, text from a publicly available corpus.

[0055] In some embodiments, in step 206, the second training dataset of the present disclosure may be constructed using a domain-specific text corpus containing conversational audio, for example, using call center conversation transcripts. In some embodiments, the conversational audio corpus may include conversations between two or more participants, characterized by multi-turn dialogues, for example, back-and-forth dialogues between a customer and an agent.

[0056] In some embodiments, a domain-specific conversational speech corpus may be obtained from recorded conversations, for example, using manual transcripts of the recorded speech conversations. In some embodiments, a domain-specific conversational speech corpus may be obtained from recorded conversations, for example, using automatic speech recognition (ASR) to recognize the recorded speech conversations.

[0057] In some embodiments, the domain-specific conversational speech corpus may be punctuated and capitalized, for example, manually. In some embodiments, the domain-specific conversational speech corpus may be annotated with corresponding punctuation and capitalization annotations, and the annotation may be performed manually by an annotation specialist.

[0058] In some embodiments, the domain-specific conversational speech corpus may include one or more of the following: • The audio may come from multiple sources, such as voice conversations, typed chats, text messaging, and email conversations. • The audio may include a conversation between at least two parties, for example, between an agent and a customer. • The audio may reflect conversations of varying lengths, and / or fragments and parts of conversations.

[0059] In some embodiments, the provided text conversational audio corpus is annotated with corresponding punctuation and capitalization annotations, and the annotation may be performed manually by an annotation specialist.

[0060] In some embodiments, in step 208, the conversational speech corpus may be preprocessed in a similar manner to the general text in the first training dataset (see above), for example, by normalizing and / or standardizing the text. For example, preprocessing may be applied to convert all words to lowercase and / or tag each word with corresponding punctuation and capitalization tags.

[0061] In some embodiments, in step 210, contextualization and / or data augmentation may be used to enhance the training data obtained from the conversational speech corpus.

[0062] In some embodiments, a conversational speech corpus can be contextualized, for example, by recognizing the fact that punctuation may be context-dependent. For instance, it is impossible to know whether the utterance "Takes a month to get there" is a question or a statement as an independent sequence. However, considering its context (e.g., preceding and / or following utterances) may reveal its purpose. The following is an example of conversational speech containing word sequences where punctuation may be context-dependent.

[0063] [Table 4]

[0064] Accordingly, in some embodiments, the disclosure provides contextualization of domain-specific conversational speech by, for example, generating conversational training segments, each containing multiple sentences. In some embodiments, such conversational speech segments may be created by, for example, segmenting a conversational speech corpus according to one or more rules. For example, when a conversation contains 12 sentences [S1, S2, ..., S12], the segmentation rule may provide segmenting the conversation into four sentence segments such that the training segments are as follows: TIFF0007870769000005.tif11153TIFF0007870769000006.tif11153E3=[S9,S 10 ,S 10 ,S 12 ]

[0065] In other embodiments, additional and / or other segmentation and / or linking rules may be applied to link, for example, more or fewer sentences to a conversational training segment.

[0066] However, a potential drawback of sentence concatenation and / or segmentation, as shown above, is that the end sentences within each conversational training segment, for example, sentence S5 in segment E2 and sentence S9 in segment E3, cannot be properly contextualized using preceding text data, while sentence S4 in segment E1 and S8 in segment E2 cannot be properly contextualized using subsequent text data, for example. (S1, of course, cannot have any preceding context).

[0067] Accordingly, in some embodiments, in step 210, the Disclosure provides data augmentation, the data augmentation mechanism being configured to expand each sentence in both directions, for example, using preceding and succeeding dialogue from a conversation. For example, the data augmentation algorithm of the Disclosure may be configured to iteratively add preceding and / or succeeding sentences to a given first sentence until the result satisfies a criterion for designating acceptability, for example, a minimum number of words and / or speakers.

[0068] In some embodiments, the data augmentation algorithms of the present disclosure may include the following:

[0069] [Table 5]

[0070] In some embodiments, the add_sentence logic is a simple logic that adds a new sentence as either a prefix or suffix to an example sentence, according to the sentence index in the conversation.

[0071] An acceptable example would be one that adheres to some specified rule, such as meeting a minimum number of words and / or speakers. For example, an acceptable example might be required to have at least two speakers and at least 25 words.

[0072] [Table 6]

[0073] Using this algorithm, we have 12 statements C=[S1,S2,...,S 12 The same conversation containing ] can be segmented here as follows: TIFF0007870769000009.tif11153TIFF0007870769000010.tif11153TIFF0007870769000011.tif11153TIFF0007870769000012.tif11153Here, the overlap between segments and the length of each segment are dynamic and determined by an algorithm, and each sentence in a conversation can be used in two or more contexts, and is usually used.

[0074] In some embodiments, in step 212, the Disclosure provides End-of-Sentence (EOS) embeddings in a training dataset. Focusing on training segments containing a single sentence, representing the input to the neural network is trivial and can be done using a standard one-hot representation, where all words are indexed in a vector by lexical size and words are input one by one in a sequence. However, when multiple sentences are included in the training examples, there is important information that may be lost, such as which is the last word of all sentences. This information is important for both punctuation and capitalization, as the last word in a sentence is almost always followed by a period or question mark, and the word that follows is always capitalized.

[0075] Accordingly, in some embodiments, the disclosure provides embedding EOS data into training examples that include multiple concatenated sentences. In some embodiments, the EOS embedding may include indications as to whether a word is "in the middle" of a sentence or "at the end" of a sentence. For example, the short dialogue presented above.

[0076] [Table 7] As a single training example for a neural network, it would look like this:

[0077] [Table 8]

[0078] The additional EOS input helps the machine learning model predict punctuation marks after the words "in" and "there," and helps the model capitalize the word "takes."

[0079] Figure 3 is a schematic example of a neural network structure that may be employed in the context of the machine learning model of this disclosure. As can be seen from the figure, the addition of EOS embeddings gives this feature considerable weight over word embeddings. In some embodiments, the embedding of the EOS feature may represent an embedding size of 30, for example, 10% of the embedding size of the word embedding. We have found that using data augmentation with EOS embeddings results in an improvement of about 10% in the classification of question marks and commas, which is the most stringent to accurately predict.

[0080] In some embodiments, in step 214, the second training dataset may be used to retrain the machine learning model of the present disclosure.

[0081] In some embodiments, in step 216, the trained machine learning model of the present disclosure may be applied to target data, for example, recognized conversational speech, to predict punctuation and capitalization of words contained in the speech.

[0082] In some embodiments, the machine learning models of this disclosure may employ a neural network structure configured for multi-task / multi-objective classification and prediction.

[0083] As background, classification tasks are typically processed one at a time. Therefore, to perform punctuation and capitalization tasks, it is typically necessary to train two sequence tagging machine learning models.

[0084] Conversely, this disclosure uses multitask learning to generate a single machine learning model trained to perform two or more tasks simultaneously. In addition to the obvious benefit of having to train only one model (offline process) and inference (online process in production), a single model also has potential informational benefits. Capitalization information used to train a capitalization-use network could theoretically contribute to punctuation training due to the strong inter-task dependency that capitalized words often follow periods. Similarly, punctuation information such as question marks and periods could train a network where the next word is capitalized.

[0085] Therefore, in some embodiments, the Disclosure employs a network architecture such as that schematically illustrated in Figure 4. In some embodiments, the exemplary neural network structure shown in Figure 4 enables the machine learning model of the Disclosure to learn punctuation and capitalization in conjunction.

[0086] In some embodiments, the Disclosure provides one or more neural network-based machine learning models trained to perform a sequence tagging task. In some embodiments, these models may include one or more long short-term memory (LSTM) networks, bidirectional LSTM networks (BiLSTM), LSTM networks with a CRF layer (LSTM-CRF), and / or bidirectional LSTM networks with a conditional random field (CRF) layer (BILSTM-CRF).

[0087] As shown in Figure 4, exemplary neural networks of this disclosure may comprise, for example, one or more of the following: a bidirectional LSTM (BiLSTM) layer, a dense layer, and / or a conditional random field (CRF) layer. In some embodiments, this disclosure may provide an exemplary neural network comprising two cooperative networks for learning capitalization and punctuation, each of which comprises, for example, one or more of the following: a BiLSTM layer, a dense layer, and a CRF layer. In some embodiments, the BiLSTM layer allows hidden states to capture both historical and future contextual information and then label the tokens. In some embodiments, the CRF layer provides considering the correlation between the current label and adjacent labels, which imposes a conditional random constraint on the outcome.

[0088] In some embodiments, the exemplary neural network architecture presented in Figure 4 provides minimizing two loss functions, e.g., a capitalization loss function and a punctuation loss function, one for each of the cooperative networks. In some embodiments, the network then computes a weighted sum of the punctuation loss and capitalization loss, representing the combined loss of the cooperative prediction. In some embodiments, the weighted sum of the separate loss functions may reflect a 2 / 3 proportion of the punctuation loss and a 1 / 3 proportion of the capitalization loss, corresponding to the relative number of classes in each task (4 and 2, respectively). Using these weights in multi-task training yields overall improvements over using separate models, in addition to any reductions in computational overhead and complexity, in both training and generation predictions.

[0089] In some embodiments, the Disclosure provides cooperative training of a machine learning model that includes a network architecture defining two cooperative networks for learning capitalization and punctuation. In some embodiments, all training segments used to train the machine learning model of the Disclosure may include two different sets of tags, namely, tags for punctuation and tags for capitalization (in addition to the actual input words and optionally EOS embeddings).

[0090] [Table 9]

[0091] In some embodiments, during the inference stage 216 in Figure 2, data augmentation may create overlaps between the inferred target speech segments, with some of the sentences appearing in multiple target segments input for inference and prediction purposes.

[0092] For example, a conversation containing four turns (or sentences) [T1, T2, T3, T4] may be used to generate two examples [T1, T2, T3] and [T2, T3, T4] for inference. In this case, for example, all words in T3 may be used twice: once in context [T1, T2, T3] and a second time in context [T2, T3, T4]. When inference is performed using a machine learning model trained on a target segment, the output may include, for example, competing predictions regarding the punctuation and / or capitalization of one or more words. In some embodiments, the trained machine learning model of this disclosure may be configured to assign a confidence score for each of the classification classes, with a total score of 1.0 for all classes. Thus, each word in this example [T1, T2, T3] will have a score for all possible tags (classes), and each word in this example [T2, T3, T4] will have a score for all possible tags (classes).

[0093] Therefore, assuming that T3 contains 5 words [w1, w2, w3, w4, w5], inferring T3 within the target segment context [T1, T2, T3] may produce the following result with respect to word w1 (for the purposes of this example, Φ indicates that it is "irrelevant" to the other words in T3).

[0094] [Table 10]

[0095] Similarly, inferring T3 within the target segment context [T2, T3, T4] may produce the following result with respect to word w1 (Φ indicates "irrelevant" to other words in T3 for the purposes of this example):

[0096] [Table 11]

[0097] Accordingly, in some embodiments, the disclosure provides a conflict tagging resolution mechanism that takes into account all predictions for every word within each target segment context. For each word, the conflict resolution mechanism averages all prediction scores received from all contexts in which it exists and ultimately selects the highest average score.

[0098] Therefore, w1 in T3 scores the average as follows:

[0099] [Table 12]

[0100] Therefore, the machine learning model output tags w1 with the punctuation tag "other," which received the highest confidence score among the possible classes.

[0101] Some embodiments of the present invention can relate to associating answers to multiple-choice questions with specific topics. For example, answers to multiple-choice questions can be compared with the question text along with topics in a manner similar to comparing the question text with various topics, in order to identify which topics distinguish these answers from others. In other words, since both the questions and answers are correlated within the dialogue document, each answer is integrated with the question to form separate question-and-answer combinations, and the resulting combinations are compared with topics to identify the most similar topics.

[0102] The present invention may be a system, method, and / or a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions for causing a processor to execute an aspect of the present invention.

[0103] A computer-readable storage medium can be a tangible device capable of holding and storing instructions for use by an instruction-executing device. A computer-readable storage medium may be, but is not limited to, electronic storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any preferred combination thereof. A non-exhaustive list of more specific examples of computer-readable storage media includes portable computer diskettes, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), compact disc read-only memory (CD-ROM), digital versatile disk (DVD), memory sticks, floppy disks, mechanically encoded devices on which instructions are recorded, and any preferred combination thereof. When used herein, computer-readable storage media should not be interpreted as being transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses passing through optical fiber cables), or electrical signals transmitted through wires. Rather, computer-readable storage media are non-transient (i.e., non-volatile) media.

[0104] The computer-readable program instructions described herein may be downloaded from a computer-readable storage medium to each computing / processing device, or to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network may include copper transmission cables, optical transmission fibers, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface within each computing / processing device receives computer-readable program instructions from the network and transfers the computer-readable program instructions for storage in a computer-readable storage medium within each computing / processing device.

[0105] The computer-readable program instructions for performing the operations of the present invention may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, state-setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the C programming language or similar programming languages. The computer-readable program instructions may run entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter scenario, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or a connection to an external computer may be made (for example, via the Internet using an Internet service provider). In some embodiments, for example, an electronic circuit including a programmable logic circuit, a field-programmable gate array (FPGA), or a programmable logic array (PLA) may execute computer-readable program instructions by personalizing the electronic circuit using state information of computer-readable program instructions in order to perform aspects of the present invention.

[0106] Aspects of the present invention will be described herein with reference to flowcharts and / or block diagrams illustrating methods, apparatus (systems), and computer program products according to embodiments of the present invention. It will be understood that each block in the flowcharts and / or block diagrams, as well as combinations of blocks within the flowcharts and / or block diagrams, can be implemented by computer-readable program instructions.

[0107] These computer-readable program instructions may be provided to the processor of a general-purpose computer, a dedicated computer, or other programmable data processing device to generate a machine such that instructions executed via the processor of the computer or other programmable data processing device create means for performing functions / actions specified in one or more blocks of a flowchart and / or block diagram. These computer-readable program instructions may also be stored in a computer-readable storage medium that can instruct computers, programmable data processing devices, and / or other devices to function in a particular manner, and as a result, the computer-readable storage medium on which the instructions are stored may include a product containing instructions that implements the modes of functions / actions specified in one or more blocks of a flowchart and / or block diagram.

[0108] Computer-readable program instructions can also be loaded onto a computer, other programmable data processing device, or other device to execute a series of operational steps on the computer, other programmable device, or other device, thereby generating a computer implementation process, the instructions executed on the computer, other programmable device, or other device implementing a function / action specified in one or more blocks of a flowchart and / or block diagram.

[0109] The flowcharts and block diagrams provided in the figures illustrate the architecture, functionality, and operation of possible implementations of the system, method, and computer program product according to exemplary embodiments of the present invention. In this regard, it will be understood that each block in the flowchart and / or block diagram may represent a module, segment, or portion of an instruction having one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions described in a block may occur in an order different from the order shown in the figure. For example, two blocks shown consecutively may actually be executed substantially simultaneously, or blocks may sometimes be executed in reverse order depending on the functions they contain. It should also be noted that each block in the block diagram and / or flowchart illustration, and combinations of blocks in the block diagram and / or flowchart illustration, may be implemented by a dedicated hardware-based system that performs a specified function or action, or a combination of dedicated hardware and computer instructions.

[0110] A numerical range should be considered to include all specifically disclosed subranges, as well as the individual numbers within those ranges. For example, a range of 1 to 6 should be considered to include specifically disclosed subranges such as 1 to 3, 1 to 4, 1 to 5, 2 to 4, 2 to 6, 3 to 6, and the individual numbers within those ranges, such as 1, 2, 3, 4, 5, and 6. This applies regardless of the width of the range.

[0111] The descriptions of various embodiments of the present invention are presented for illustrative purposes only and are not intended to be exhaustive or limitful to the disclosed embodiments. Many modifications and variations will be obvious to those skilled in the art without departing from the scope and spirit of the embodiments described. The terms used herein have been selected to best describe the principles, practical applications, or technical improvements to the technologies available on the market of the embodiments, or to enable other those skilled in the art to understand the embodiments disclosed herein.

[0112] The experiments conducted and described above demonstrate the usefulness and effectiveness of embodiments of the present invention. Some embodiments of the present invention may be constructed based on specific experimental methods and / or experimental results, and therefore the following experimental methods and / or experimental results should be considered embodiments of the present invention.

Claims

1. It is a system, At least one hardware processor, A non-temporary computer-readable storage medium storing program instructions, The program instructions are performed by the at least one hardware processor. Receiving a first text corpus containing punctuation and capitalized text, annotating the words in the first text corpus with a set of labels, the labels indicating the punctuation and capitalization associated with each of the words in the first text corpus, and training a machine learning model on a first training set in an initial training phase, the first training set being (i) The annotated word in the first text corpus, (ii) Training, including the labels, receiving a second text corpus representing conversational speech, annotating words in the second text corpus with the set of labels, wherein the labels indicate punctuation and capitalization associated with each of the words in the second text corpus, and in a retraining stage, retraining the machine learning model on a second training set, the second training set being (iii) The annotated word in the second text corpus, (iv) A system capable of retraining, including the labels, and in the inference stage, applying the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

2. The system according to claim 1, wherein the label indicating punctuation is selected from the group consisting of commas, periods, question marks, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

3. The system according to claim 1, wherein the first text corpus is preprocessed before training by converting at least all words in the first text corpus to lowercase.

4. The system according to claim 1, wherein the second text corpus is preprocessed by performing contextualization before the retraining, the contextualization comprising segmenting the text corpus into segments, each containing at least two sentences.

5. The system according to claim 4, wherein the second text corpus is preprocessed before the retraining by performing data augmentation, the data augmentation comprising extending at least some of the segments by adding at least one of one or more preceding sentences in the conversational audio and at least one of one or more following sentences in the conversational audio.

6. The system according to claim 4, wherein the prediction includes a confidence score associated with each of the predicted punctuation and predicted capitalization, and when a word in the target set is included in two or more of the segments and two or more of the predictions regarding punctuation or capitalization are received, the system averages the confidence scores associated with the two or more predictions to produce a final confidence score for the prediction.

7. The system according to claim 1, wherein the second text corpus is preprocessed by including end-of-sentence (EOS) embeddings before the retraining.

8. The system according to claim 1, wherein the second text corpus and the target set of words each include transcribed text representing a conversation between at least two participants, the at least two participants being a call center agent and a customer.

9. The system according to claim 8, wherein the transcription includes at least one analysis selected from the group consisting of text detection, speech recognition, and speech-to-text detection.

10. A method performed by a system, Receiving a first text corpus containing punctuation and capitalized text, annotating the words in the first text corpus with a set of labels, the labels indicating the punctuation and capitalization associated with each of the words in the first text corpus, and training a machine learning model on a first training set in an initial training phase, the first training set being (i) The annotated word in the first text corpus, (ii) Training, including the labels, receiving a second text corpus representing conversational speech, annotating words in the second text corpus with the set of labels, wherein the labels indicate punctuation and capitalization associated with each of the words in the second text corpus, and in a retraining stage, retraining the machine learning model on a second training set, the second training set being (iii) The annotated word in the second text corpus, (iv) A method comprising retraining a model including the labels, and in the inference stage applying the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

11. The method according to claim 10, wherein the label indicating punctuation is selected from the group consisting of commas, periods, question marks, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

12. The method according to claim 10, wherein the first text corpus is preprocessed before the training by converting at least all words in the first text corpus to lowercase.

13. The method according to claim 10, wherein the second text corpus is preprocessed by performing contextualization before the retraining, the contextualization comprising segmenting the text corpus into segments, each containing at least two sentences.

14. The method according to claim 13, wherein the second text corpus is preprocessed by performing data augmentation before the retraining, the data augmentation comprising extending at least some of the segments by adding at least one of one or more preceding sentences in the conversational audio and at least one of one or more following sentences in the conversational audio.

15. The method according to claim 13, wherein the prediction includes a confidence score associated with each of the predicted punctuation and predicted capitalization, and when a word in the target set is included in two or more of the segments and two or more of the predictions regarding punctuation or capitalization are received, the confidence scores associated with the two or more predictions are averaged to produce a final confidence score for the prediction.

16. The method according to claim 10, wherein the second text corpus is preprocessed by including end-of-sentence (EOS) embeddings before the retraining.

17. A computer program including program instructions, wherein the program instructions are performed by at least one hardware processor. Receiving a first text corpus containing punctuation and capitalized text, annotating the words in the first text corpus with a set of labels, the labels indicating the punctuation and capitalization associated with each of the words in the first text corpus, and training a machine learning model on a first training set in an initial training phase, the first training set being (i) The annotated word in the first text corpus, (ii) Training, including the labels, receiving a second text corpus representing conversational speech, annotating words in the second text corpus with the set of labels, wherein the labels indicate punctuation and capitalization associated with each of the words in the second text corpus, and in a retraining stage, retraining the machine learning model on a second training set, the second training set being (iii) The annotated word in the second text corpus, (iv) A computer program executable to retrain, including the labels, and in the inference stage, to apply the trained machine learning model to a target set of words representing conversational speech to predict the punctuation and capitalization of each word in the target set.

18. The computer program according to claim 17, wherein the first text corpus is preprocessed before training by converting at least all words in the first text corpus to lowercase.

19. The computer program according to claim 17, wherein the label indicating punctuation is selected from the group consisting of commas, periods, question marks, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

20. The computer program according to claim 17, wherein the second text corpus is preprocessed before the retraining by performing at least one of the following: contextualization, which includes segmenting the text corpus into segments, each containing at least two sentences; data augmentation, which includes extending at least some of the segments by adding at least one of one of one or more preceding sentences in the conversational audio and at least one of one or more following sentences in the conversational audio; and end-of-sentence (EOS) embedding.

21. It is a system, At least one hardware processor, A non-temporary computer-readable storage medium storing program instructions, The program instructions are performed by the at least one hardware processor. It is executable to perform the operation of a multitasking neural network, and the multitasking neural network is A capitalization prediction network that takes a text corpus containing at least one sentence as input and predicts the capitalization of each word in the at least one sentence, the capitalization prediction network being trained based on a first loss function, A punctuation prediction network that receives the aforementioned text corpus as input and predicts punctuation on the aforementioned text corpus, the punctuation prediction network being trained based on a second loss function, A system comprising: an output layer that outputs coordinated predictions of capitalization and punctuation based on a multitask loss function combining the first and second loss functions, wherein the capitalization prediction network and the punctuation prediction network are trained in cooperation.

22. The system according to claim 21, wherein the program instruction can be further executed in the inference stage to apply the multitask neural network to a target set of words representing conversational speech to predict punctuation and capitalization of each word in the target set.

23. The aforementioned coordinated training includes, in the initial training stage, training the capitalization prediction network and the punctuation prediction network in coordination with respect to a first training set, wherein the first training set is (i) A first text corpus including punctuated and capitalized text, (ii) a label indicating punctuation and capitalization associated with each of the words in the first text corpus, the system according to claim 21.

24. The aforementioned coordinated training further includes, in the retraining phase, training the capitalization prediction network and the punctuation prediction network in coordination with a second training set, wherein the second training set is (iii) A second text corpus representing conversational audio, (iv) a label indicating punctuation and capitalization associated with each of the words in the second text corpus, the system according to claim 23.

25. The system according to claim 24, wherein the label indicating punctuation is selected from the group consisting of commas, periods, question marks, and others, and the label indicating capitalization is selected from the group consisting of capitalization and others.

26. The system according to claim 24, wherein the first text corpus is preprocessed before the training by converting at least all words in the first text corpus to lowercase.

27. The system according to claim 24, wherein the second text corpus is preprocessed by performing contextualization before the retraining, the contextualization comprising segmenting the text corpus into segments, each containing at least two sentences.

28. The system according to claim 27, wherein the second text corpus is preprocessed before the retraining by performing data augmentation, the data augmentation comprising extending at least some of the segments by adding at least one of one or more preceding sentences in the conversational speech and at least one of one or more following sentences in the conversational speech.

29. The system according to claim 24, wherein the second text corpus is preprocessed by including end-of-sentence (EOS) embeddings before the retraining.

Images