Methods, apparatus, and computer programs for extracting and generating nonverbal messages

Abstract

A method and apparatus comprising computer code configured to cause one or more processors to receive text containing multiple sentences, to cause a machine learning model to extract a non-verbal message from one of the sentences, to add annotations to the text, the annotations to indicate the non-verbal message, and to output a version of the text containing the annotations.

Description

[Technical Field]

[0001] [Related applications] This application claims priority under U.S. Patent Application No. 18 / 384428, filed October 27, 2023, the entirety of which is incorporated herein by reference.

[0002] [Technical field] This disclosure relates to the extraction and generation of nonverbal messages. [Background technology]

[0003] Nonverbal messages (NM), such as a speaker's facial expressions and speaking speed, are essential for face-to-face communication and can be considered tacit knowledge that is not typically included in existing dialogue comprehension and generation tasks.

[0004] Previous research has focused on extracting non-narrative texts (NMs) from relatively small, structured corpora such as movie scripts. However, in this corpus, the NMs are enclosed in parentheses by the scriptwriters, significantly reducing the difficulty of extraction.

[0005] One of the most relevant text resources for nonverbal messages is television and movie scripts. Generally, scripts are written in a standard format. For example, the nonverbal messages (NM) of corresponding utterances are enclosed in parentheses (e.g., ELIZABETH (ironically) "With an annual salary of 5,000, it wouldn't matter if he had a big pink face," and MR DARCY (shakes head) "You know how much I hate it," which usually describes what the viewer can see and hear beyond the linguistic message. Based on a clearly defined movie script structure, it is relatively easy to extract utterances and their NMs from scripts using heuristics. However, in scripts, only a small percentage of utterances (about 10.5% based on the authors' analysis) are followed by NMs, and existing public script corpora are usually small, even for resource-rich English (e.g., 1276 movies, 917 films).

[0006] Previous research has designed patterns or language-specific features to identify NMs, utterances, and speakers from well-structured (or semi-structured) scripts that can be extracted relatively easily based on film script format. To the authors' knowledge, no studies have investigated the automated extraction of NMs from unstructured corpora, and therefore, existing computer technologies are simply unable to perform such feature extraction, even if required.

[0007] NMs such as facial expressions and body postures are used to facilitate tasks such as classifying dialogue behaviors, deception detection, and text-to-speech. However, because these messages are either predefined or transformed based on recorded video, the resources required to collect the data are consumed, making it difficult to scale up to other applications. NMs extracted from scripts have been shown to be useful for tasks requiring dialogue comprehension, such as machine reading comprehension and relation extraction. However, there are no existing uses for NMs extracted from unstructured data for both comprehension and generation tasks, and existing computer technologies simply cannot implement such features, even if required. There are no publicly available datasets in any language for identifying NMs in unstructured corpora.

[0008] And for any of these reasons, there is a need for technical solutions to such problems that have arisen in computer audio technology. [Overview of the Initiative]

[0009] The present invention includes a method and apparatus comprising a memory configured to store computer program code, and one or more processors configured to access the computer program code and operate as directed by the computer program code. The computer program is transmitted to the processors, A receiving code configured to cause at least one processor to receive text containing multiple sentences, An extraction and annotation code configured to cause at least one processor to extract a nonverbal message from one of the sentences using a machine learning model and to add an annotation to the text, wherein the annotation represents the nonverbal message and includes the extraction and annotation code. Output code configured to cause at least one processor to output a version of the text including the annotations, It is configured to carry out the following actions.

[0010] Extracting the non-verbal message may include determining the type of the non-verbal message, and the type is at least one of a motion type, an internal state type, a rest type, and a voice type.

[0011] The motion type may indicate at least one of body movements other than expressions and expressions. The internal state type may indicate at least one of intention and emotion. The rest type may indicate the passage of time. The voice type may indicate at least one of a listener, the number of speakers, tone, volume, speed, pitch, and timbre.

[0012] Extracting the non-verbal message may include determining the context of the one sentence based on determining the contexts of a preset number of sentences both before and after the one sentence by the machine learning model.

[0013] The extraction of the non-verbal message may be based on the minimization of the following formula.

Number

Number

[0014] , , , , start , start , and p​​​​​​​​​​​​​ Here, y t x represents the nonverbal message of the above sentence, t θ represents the context within a predetermined distance of the aforementioned sentence, e and θ d This represents the parameters of the machine learning model.

[0015] The above annotation may be based on the following probability determination:

number

[0016] The step of receiving the text may include the step of inputting at least one of a movie script and a novel into the machine learning model.

[0017] The at least one sentence may indicate that the speaker uttered a conversation, and the annotation may indicate that the conversation was uttered as a shout.

[0018] The aforementioned text may include multiple speakers. [Brief explanation of the drawing]

[0019] Further features, characteristics, and various advantages of the subject matter of disclosure will become clearer from the detailed description and accompanying drawings below.

[0020] [Figure 1] This is a schematic diagram of a computer system according to an embodiment.

[0021] [Figure 2] This is a simplified block diagram of an embodiment.

[0022] [Figure 3] This is a simplified flowchart of an embodiment.

[0023] [Figure 4] This is a simplified diagram of an embodiment.

[0024] [Figure 5] This is a simplified diagram of an embodiment.

[0025] [Figure 6] This is a simplified diagram of an embodiment.

[0026] [Figure 7] This is a simplified diagram of an embodiment.

[0027] [Figure 8] This is a schematic flowchart of an embodiment.

[0028] [Figure 9] This is a simplified diagram of an embodiment.

[0029] [Figure 10] This is a simplified diagram of an embodiment.

[0030] [Figure 11] This is a simplified diagram of an embodiment.

[0031] [Figure 12] This is a simplified diagram of an embodiment.

[0032] [Figure 13] This is a simplified diagram of an embodiment.

[0033] [Figure 14] This is a simplified diagram of an embodiment.

[0034] [Figure 15] This is a simplified diagram of an embodiment.

[0035] [Figure 16]This is a simplified flowchart of an embodiment.

[0036] [Figure 17] This is a simplified flowchart of an embodiment.

[0037] [Figure 18] This is a simplified flowchart of an embodiment.

[0038] [Figure 19] This is a simplified flowchart of an embodiment.

[0039] [Figure 20] This is a simplified diagram of an embodiment. [Modes for carrying out the invention]

[0040] The proposed features described below may be used separately or combined in any order. Furthermore, embodiments may be implemented by processing circuits (e.g., one or more processors or one or more integrated circuits). In one example, one or more processors execute a program stored in a non-temporary computer-readable medium.

[0041] Figure 1 shows a simplified block diagram of a communication system 100 according to an embodiment of the present disclosure. The communication system 100 may include at least two terminals 102 and 103 interconnected via a network 105. In one-way data transmission, the first terminal 103 may encode video data at its local location for transmission to the other terminal 102 via the network 105. The second terminal 102 may receive the encoded video data from the other terminal via the network 105, decode the encoded data, and display the restored video data. One-way data transmission may be common in media serving applications, etc.

[0042] Figure 1 shows a second pair of terminals 101 and 104 provided to support bidirectional transmission of coded video, which may occur, for example, during a video conference. In bidirectional data transmission, each terminal 101 and 104 may code locally captured video data for transmission to other terminals via the network 105. Each terminal 101 and 104 may also receive coded video data transmitted by other terminals, decode the coded data, and display the restored video data on a local display device.

[0043] In Figure 1, terminal devices 101, 102, 103, and 104 may be shown as a server, a personal computer, and a smartphone, but the principles of this disclosure are not limited to these. Embodiments of this disclosure include applications by laptop computers, tablet computers, media players, and / or dedicated video conferencing equipment. Network 105 represents any number of networks that carry coded video data between terminal devices 101, 102, 103, and 104, including, for example, wired and / or wireless communication networks. Communication network 105 may exchange data over circuit switching and / or packet switching channels. Typical networks include electronic communication networks, local area networks, wide area networks, and / or the Internet. For the purposes of discussing the present invention, the architecture and topology of network 105 may not be important to the operation of this disclosure unless otherwise specified below.

[0044] Referring to Figure 2, a block diagram of system 200 according to an exemplary embodiment is shown. Figure 2 can be explained using the exemplary embodiment shown in Figure 1. According to one or more embodiments, system 200 can be deployed in a computer, a server, a collection of computers, a cloud computing environment, etc. Thus, system 200 can include, among other things, a preprocessing module 202 and a deep neural network 204. The preprocessing module 202 may include a digital signal processing (DSP) module 208 and can be configured to retrieve data 206. According to one embodiment, data 206 can be retrieved from a data storage device. In an alternative embodiment, data 206 can be retrieved from a database 112 (Figure 1) on a server computer 114. Data 206 can include, among other things, raw ECG data collected from patients. According to one embodiment, data 206 can be a complete 24-48 hour long collection period. According to an alternative embodiment, data 206 can be a random sample of the collection period. According to yet another alternative embodiment, data 206 can be a sample of the collection period having the highest variance value. The DSP module 208 can extract one or more RR intervals from the data 206 by segmenting the data for each individual heartbeat. This can be achieved, for example, by calculating the time interval between consecutive R-wave peaks. Thus, the DSP module 208 can, among other things, assist in converting the one-dimensional time signal corresponding to the ECG data into a multi-dimensional array for processing by the deep neural network 204. The DSP module 208 can also apply data cleaning and filtering to the data 206 to improve processing by the deep neural network 204.

[0045] The deep neural network 204 may include, among other things, an input matrix 210, one or more hidden layers 212, 214, and 218, a feature transformation layer 216, a pooling layer 220, and one or more connected layers 222 and 224. Figure 2 shows only one implementation of the deep neural network 204, and it should be understood that the deep neural network 204 is not limited to these exact layers and their order. The deep neural network 204 may include any number of layers in any order, including adding or omitting any of the layers shown.

[0046] The input matrix 210 can be, for example, a two-dimensional matrix with dimensions n × k, where n is the number of RR intervals (i.e., heart rates) selected for analysis, and k-1 may be the number of previous RR intervals for each of the RR intervals. For example, if 128 RR intervals are analyzed using a lookback window of three previous RR intervals for each of the 128 RR intervals, the input matrix will have a size of 128 × 4. However, n and k may be arbitrary values ​​that can be selected based on available computing power, and it can be understood that for larger k values, more neighbor information may be preserved for each heartbeat.

[0047] A feature transformation layer 216 may be used to extract one or more features. Although only one feature transformation layer 216 is shown, it should be understood that the deep neural network 204 may include additional feature transformation layers 216 that can be applied to the data 206 in series or in parallel. One or more hidden layers 212, 214, and 218 may be used to further process the data into a format usable by the deep neural network 204. A pooling layer 220 may be used to aggregate one or more features and to downsample the analyzed data to facilitate the identification of one or more features. The pooling layer 220 may apply a max pooling strategy, a mean pooling strategy, or other pooling methods. A first fully connected layer 222 may be used, for example, to classify the aggregated features and compare the features to one or more patterns. Patterns can be developed by deep learning so that there is no human intervention in the creation of the patterns. A second fully connected layer 224 may be used to classify whether the data 206 contains patterns by analyzing the output of the first fully connected layer 222. The second fully connected layer 224 can apply an instruction function to the data, for example, by outputting "1" if the data contains a pattern and "0" if the data does not contain a pattern.

[0048] In the exemplary embodiment, since there are no public datasets for identifying nonverbal message extractions (NMs) in unstructured corpora in any language, we first introduce NM annotation guidelines and annotated datasets for nonverbal message extraction (NME) analysis.

[0049] For an exemplary embodiment, see Example 300 in Figure 3. In Figure 3, the annotation task is designed to simplify the annotation task by copying the writer's words and to avoid potentially inaccurate interpretations by the annotator, by selecting one or more spans (which may not be adjacent) from the text surrounding a given utterance in S302, or obtaining them in S301. Because several NMs may exist in the context, in S303, the embodiment may retain only those that occur shortly before or simultaneously with the utterance being spoken (within a predetermined number of phrases or sentences, such as five phrases or sentences before or after the target, but this number is adjustable depending on the embodiment). The speaker of the utterance may not be included in the selected spans. Furthermore, the embodiment includes three common phonetic verbs.

number

number

[0050] Accordingly, as will be further discussed herein, embodiments construct unannotated NM data, such as in Example 401, based on a public speaker identification dataset that includes novels, where the number of novels may be three in the case of Example 400 shown in Figure 4. Embodiments can then use the annotated target utterance, its labeled speaker, and the context of 10 sentences (5 sentences before / after the utterance) without modification and follow data partitioning. Since "speaker:utterance" is considered as two sentences separated by ":" in the dataset, the actual number of sentences in the context may be less than 10. Each instance (Example 400) is annotated by two annotators, all of whom are native Chinese speakers.

[0051] Example 400 demonstrates, according to an embodiment, that an annotated NME dataset can be achieved when annotations 403 and 404 are added to Example 402, compared to the original text in Example 401.

[0052] Novel authors tend to omit nonverbal notes (NMs) when the utterance or context is self-evident. Furthermore, instances with relatively low informational NMs are discarded. Thus, the embodiment can retain only 67.4% of annotated utterances, with each instance corresponding to one or more nonverbal messages. This is evident from the NME statistic 500 shown in Figure 5, according to the exemplary embodiment, and improves computational efficiency.

[0053] To examine the detailed types of NMs, the embodiment reviews the literature, analyzes annotated spans, and ultimately classifies them into 13 subtypes, as shown in Example 600 of Figure 6, according to the exemplary embodiment. NMs can belong to multiple subtypes. For example, the following shows both facial expression and intent:

number

number

number

[0054] According to an exemplary embodiment, there is an NME pattern-based method that can achieve S304 annotation and S305 training by removing speaker names and utterances from consideration when S302 extracts NMs. Since the writer's own observations, which may contain NMs, are interspersed with the utterances of characters, the embodiment can assume that NMs are very likely to appear in the same paragraph as a given target utterance. Thus, the embodiment can first run a robust (approximately 90% at F1) extracted speaker identification model on the paragraph to identify the speaker of the target utterance. Since utterance annotations are not available in unlabeled novels, the embodiment uses double quotes to segment the utterance, considering the first one as the target utterance. Next, the embodiment can reduce noise by removing all utterances and speakers from the paragraph, separating the remaining context with commas, and using the last span as the NM for the target utterance. For example, if there is a paragraph like the one in Figure 4, “Miejue Shitai shouted “Demon Cult!””, the embodiment relating to a pattern-based extractor would extract “shouted” as the NM for the underlined utterance. However, these embodiments may inevitably suffer from relatively low recall rates (for example, the macro-average recall rate for the NME dev set is 63.1%). For example, given the context of E6 in Example 600, this method can only extract "thought," but both "heart was relieved" and "thought" should be considered NM.

[0055] Therefore, the embodiments also use an extraction method. Since the reference to the non-verbal message must be one or more spans in the context surrounding the target utterance, the embodiments consider a machine reading comprehension (MRC) formulation originally aimed at extracting answers to a given question from a document. Referring to example 800 in FIG. 8, in S801, the embodiments regard the target utterance u as a question, and in S802, regard u and the context around u as the document d. The grand truth non-verbal message of u is treated as the answer a in S803. In S804, the embodiments concatenate the special token [CLS], the tokens within u, the special token [SEP], and the tokens within d as an input sequence. The [SEP] token can indicate when the next sentence starts for the next sentence prediction (NSP) task. The [CLS] token can be added to sequence A and sequence B to form the input, and the target of [CLS] is whether sequence B actually follows sequence A in the corpus. In S805, two vectors p start and p end are introduced, and the estimated probability of each token in d represents that each is the start token or end token of the correct span a that appears in d. a start and a end shall indicate the start offset and end offset of a respectively. In S806, the embodiments, hereinafter

Number

Number

[0056] However, embodiments using such classical extraction architectures can only extract a single span from a context, although there are several extensions for extracting multiple spans. For these instances with multiple NMs, embodiments can use the longest common substring of the context and the concatenation of these NMs as the actual answer for training in S305 and as the original label for validation. To further improve these embodiments, multi-span formulations are also described further below.

[0057] For example, to address the single-span limitation of the extraction method described above, the embodiment also uses a generation method. In this method, nonverbal message extraction can be considered a text-to-text task, and the extractor is given the context surrounding a target utterance S802 and asked to generate an NM for this utterance. In S808, for the NM, which is a set of discontinuous spans, the embodiment concatenates them using commas in S809 to form a ground truth label. Furthermore, in S810, the broken objective function:

number

number

[0058] In the input sequence, preceding / following contexts can be separated by special tokens [SEP]. Embodiments were experimented with different types of input, as shown in Example 1000 in Figure 10. Despite the advantage of extracting multispan messages, the generation method may have hallucinogenic problems compared to faithful NMs extracted by pattern-based extraction methods.

[0059] Therefore, according to embodiments in which the aforementioned pattern-based extractor and speaker identification model are implemented to extract (context, utterance, NM) candidates with non-empty NMs from hundreds of novels, there may be large-scale weak data construction. For data quality control, embodiments discard instances where the NM exceeds 9 characters or is relatively uninformative. If the paragraph preceding the target utterance also contains an utterance, embodiments note that they use the speaker and the last utterance (separated by a colon) as context to match the data format to the script format. If there is no utterance in the preceding paragraph, the entire content of the paragraph is used as context. To obtain NMs extracted by trained extractors and generators, which are used to replace pattern-based NMs in (context, utterance, NM) triples, embodiments use the paragraph containing the target utterance and the preceding paragraph as context for inference. In summary, embodiments construct the same number of (context, utterance, NM) triples using each of the three extraction methods introduced in this section to extract the corresponding NMs.

[0060] Furthermore, according to an exemplary embodiment, there is NM generation, which in training S305 and annotation S304 involves k utterances or narratives:

number

number

number

[0061] Similar to the generative extraction method described above, such an embodiment employs the text-to-text formulation described above and, in S305, trains the generator with the highest probability of predicting the target sequence.

[0062] Such a training framework using automatically constructed data leverages the weakly labeled nonverbal message generation instance constructed by the extraction method described above through a two-stage fine-tuning in S807. Here, the embodiment first fine-tunes the generator for a combination of clean and weakly labeled data, and then fine-tunes the resulting generator for clean data only. Such a framework is shown in Example 900 of Figure 9, which demonstrates a supervised extractor trained with NME.

[0063] In light of the above, and regarding the extracted baseline, the embodiments include encoders only (RoBERTa-wwm-ext-large and MacBERT, etc.) and encoder-decoder models (T5 base BART large , and DialBART large Consider both (etc.). DialBART large As explained below, LUGE dialogue BART largeIt should be noted that this is obtained by fine-tuning. The above encoder-decoder model is also used in a generated NM extractor and NM generator according to an embodiment that can also discard tokens from the bottom of the input if its length exceeds the maximum model sequence length. Experiments according to the exemplary embodiment were performed on eight NVIDIA-V100 32GB GPUs. Each experiment was run five times with a different random seed, and the results demonstrating the effectiveness of the above exemplary embodiment are provided below.

[0064] Furthermore, as part of the experiment, Chinese scripts were used to construct clean NM generation data. Scripts were collected from script websites, and after filtering out those with formatting issues, 454 scripts were retained. To avoid data leakage, triples extracted from non-duplicate scripts were used as the training and development sets. The development set consisted of the most recent 50 scripts.

[0065] Furthermore, to construct weakly labeled NM generation data, 521 Chinese novels were collected, and the experimental results yielded two weak NM generation datasets of different sizes: Novel (397K) and Novel L (749K) is provided along with Novel. L It is a subset of.

[0066] Furthermore, a common-sense knowledge approach was also employed. Here, the context-utterance-NM triple is considered common-sense knowledge because both are usually ignored and assumed to be known without being told. Thus, the embodiment used human-annotated argument pairs from the set of common-sense knowledge in Chinese ("Someone has an umbrella," "It's raining") by considering the two arguments as the input and NM, respectively. The context was left empty.

[0067] Furthermore, LUGE dialogueThis includes four Chinese dialogue datasets: Chinese Persona Chat (CPC), LCCC, Emotional STC (ESTC), and KdConv. The embodiment uses DialBART, fine-tuned with this dataset, as a backbone model to examine the usefulness of dialogue generation datasets for NM generation. large Using LUGE indirectly dialogue I used it.

[0068] Furthermore, C 3 Regarding EWECT: To investigate the impact of introducing generated NMs into dialogue tasks without NMs, a multi-choice MRC dataset C was used for dialogue / narrative comprehension tasks. 3 We also considered the emotion classification dataset EWECT (general domain version).

[0069] Example 1000 in Figure 10 shows the results of an NM extraction evaluation in which the supervised extractor is clearly superior to the pattern-based method. This shows that including the target utterance in the input degrades the performance of the generation method, even when the utterance boundary is indicated by [SEP]. Methods such as increasing the training data size (clean or noisy) of the NME according to the embodiment help the model learn to focus on the writer's words to identify NMs. The distribution of NM lengths extracted by the supervised extractor of the embodiment is very similar to the distribution of clean NM lengths in the script, as shown in Example 1100 in Figure 11. Note that the sharp decline in the pattern-based NM distribution is caused by length constraints set for weak data construction. Example 1100 shows the nonverbal message extraction performance for the NME dataset (cat: concatenation, context1: context before the target utterance, context2: context after the target utterance, EM: exact match) and the distribution of nonverbal message lengths extracted by the script and the three methods of the embodiment.

[0070] For the experimental results of the NM generation method in this embodiment, for the majority of baselines, the following was used as the NM for all utterances, which was the most frequent NM (1.03%) in the script training set:

number

[0071] Example 1200 in Figure 12 shows the average performance and standard deviation of non-verbal generation in the script development set (->>: 2-stage fine-tuning, *: DialBART). large LUGE dialogue (Because it has been fine-tuned in advance.)

[0072] Furthermore, to consider the influence of context and speaker ID and to investigate the impact of context on NM generation, the context was removed from the training instances. In other words, the input is now only the target utterance. T5 is used in ablation research. base We used the following. Based on the input sequence, in our main experiment we further added an earlier context (i.e., one story or utterance) or a target utterance u k The speaker(s) (multiple speakers) were used and added. As shown in Example 1300 in Figure 13, the context before the target utterance is important for NM generation, but more historical context (u k-2 ) or u k Introducing speakers (multiple speakers) does not lead to a significant performance improvement. Furthermore, the results suggest that other dialogue-related datasets without speaker information can also be considered for use in improving NM generation.

[0073] Example 1300 is automatically constructed data (Novel generative )(u k :k-th utterance, i.e., the target utterance, s k This represents the results of a study on the impact of introducing contextual and speaker information for semi-supervised training based on the speaker of the target utterance (different components of the input are separated by [SEP]).

[0074] Furthermore, human evaluation of the results of this embodiment was also considered. 100 instances were randomly sampled from the presented set of scripts, and for each instance, the label and the auto-generated NM were randomly shuffled. Given context, target utterance, and NM, human commentators were asked to evaluate each NM using the following four binary metrics: (M1) relevance between the utterance and NM based on context, (M2) fluency of the NM, (M3) validity of the NM, and (M4) factual consistency of the NM based on context and utterance. For human evaluation of NM generation, human agreement (κ) was measured using Cohen's kappa (κ coefficient). For all four metrics, κ = 0.55 (moderate agreement). When the hallucinogenic problem in M4 is not considered, and this metric cannot be used to determine the ground truth NM label, κ = 0.64 (substantial agreement). Similar to observations made with automated metrics, as reported in Example 1400 of Figure 14, models trained on automatically extracted data achieve better performance than purely supervised baselines trained with scripts.

[0075] Example 1400 in Figure 14 shows the human evaluation (%) of the retained set of scripts (M1: relevance, M2: fluency, M3: validity, M4: consistency, AVG4 / 3: average of M1-4 and M1-3).

[0076] Furthermore, aspects of evaluating natural language comprehension tasks were also considered, and it was studied whether the predicted NM (from different generators in Example 1200) was useful for dialogue / narrative comprehension tasks. A pre-released baseline was adopted, and for each utterance in the input of each instance, the predicted NM was added after the utterance without changing any other settings. For example, C aimed at selecting the correct answer option for a question based on the dialogue. 3 DOne example of the revised dialogue (English translation) is as follows: "Woman: Hey, where are you? We're all waiting for you! (shouting urgently) Man: Right away, I'm coming right away! I've already gotten off the bus and I'm on my way to you! (said urgently)." The same procedure was followed for EWECT, which aims to identify the emotional states of the writer and speaker. As shown in Example 1500 in Figure 15, the significant improvement achieved by introducing NM into the original task is achieved in a human-interpretable manner compared to the same implementation baseline without considering NM, and NM has a similar impact to clean common sense and script knowledge for the two tasks.

[0077] Example 1500 in Figure 15 shows NM to C 3 D And represents the accuracy (%) to be introduced into EWECT:

number

[0078] Figure 16 shows in detail one embodiment of the text normalization device 1600. The text normalization device 1600 includes input text 1610, which is fed to a Bidirectional Encoder Representations from Transformers (BERT) 1620 from a character-based transformer. The output of the BERT 1620 is then actuated by a rule-based model 1630. The rule-based model 1630 further includes a segment converter 1640, a position switcher 1650, a numerical converter 16560, and a metric / mark substitute 1670. After processing in the tag-based converter 1630, the normalization device 1600 finally outputs normalized text 1680. The text normalization device 1600 may include additional rule layers, BERT modules of different sizes, and additional blocks such as additional checks and calculations.

[0079] Input text 1610 includes a string of text consisting of combinations of characters including text and other markers such as punctuation. For example, input text 1610 may be a string such as a date. A date typically includes a mixture of markers and text. There are various forms that contain the same information, such as "February 2, 2014" or "2 / 2 / 14". Other examples may be a text message that typically includes other markers such as emojis, or an email that includes various punctuation marks such as the "@" symbol. Another example of input text 1610 may be a metric or measurement, which may be ambiguous and without context. For example, "16M" may mean "16 megabytes" or "16 meters" depending on how it is read. Finally, as a non-exhaustive list, input text 1610 may be mathematics, a website URL, program code, language text, a combination of languages, any combination of the examples above, or any unique marker.

[0080] After receiving the input text 1610, the text normalization machine 1500 passes the text to a character-based BERT 1520. The character-based BERT 1620 is fully trained and labels each character with predefined tags. In some embodiments, BERT 1620 is a full-size BERT, and in other embodiments, it is a single-layer BERT model. Single-layer BERTs are generally faster than full-size BERTs, but smaller modules can result in performance degradations such as a higher error rate. One way to compensate for this degradation is to add a Conditional Random Field (CFR) loss function that labels entire segments instead of individual characters. Other embodiments may use a full-size BERT with a CFR loss function, a single-layer BERT without CFR, or a full-size BERT without a CFR loss function.

[0081] After tagging the input text, BERT1620 passes that information to the rule-based model 1630. The rule-based model 1630 receives the tagged text, parses the text, and finally creates a normalized text output 1680. Inside the rule-based model, there are two stages: the first stage includes a segment merger 1640 and a position switcher 1650, and the second stage includes a numerical converter 1660 and a metric / mark substitute 1670. To normalize the text, the rule-based model passes the tagged input text to each stage, which contains various rules.

[0082] The text first goes through each rule of the first stage. For convenience, we will explain the segment merger 1640 first. The segment merger 1640 merges similar characters while preserving their meaning. For example, when converting a date, the merger takes separate strings such as "February", "2", and "2014", merges them to form 2014 / 02-02, and ensures that the text can be properly read by a machine or other computer device.

[0083] The position switcher 1650 switches the position of the input string so that, when read, the normalized text places the words in the correct order. For example, in languages ​​that indicate the type of thing before the value, such as the Chinese "12.5%" where the percent sign is pronounced before the number, the position switcher 1650 switches to read the percent sign before the number. Switching can also be performed on any text where the meaning is better preserved by changing the position of characters or words within the input text.

[0084] After passing through the first stage, the text string is processed by a second stage, which includes a numeric converter 160 and a metric / mark substitute 1670. The numeric converter 16560 converts numbers to text. For example, the input may be the string "48", and after the numeric converter converts the string to text, the string reads "forty-eight" as output. The metric / mark substitute 1670 searches the text for punctuation marks or other commonly used abbreviations or other random symbols / characters and replaces them with word expressions. As an example, the metric / mark substitute 1670 takes the input string "&" and replaces it with "and" as output. Other metrics / marks can include emojis, slang, symbols, hieroglyphs, diverse spellings, ASCII art, or other emoji items that convey meaning. Finally, after the string has been fully processed, it is output as normalized text 1680.

[0085] Figure 17 details the phrase-based attention 1700 and the process of determining phrase boundaries as part of the input to BERT. The phrase-based attention 1700 consists of input text 1710, BERT embedding 1720, embedding string 1730, phrase embedding 1740, BERT encoding 1750, and restricted output 1760. As an example, input text 1710 contains the date "2018 / 01-09" for processing. As mentioned above, input text 1710 can be any text or other emoji material. Next, input text 1710 is passed to BERT embedding 1720, which is labeled for processing. In BERT embedding 1720, labels are defined as 0 if not a boundary and 1 if it is the end of a phrase. For example, "2018" has the label "0001".

[0086] After the embeddings are calculated, the character embeddings are replaced with phrase embeddings 1740, which are the average of all character embeddings within this phrase, with or without additional weights for those characters. Here, for example, the phrase "2018 / 01-09" receives its weight in the calculation shown in Figure 17. After the character embeddings have been processed, the next part is BERT encoding 1750. Finally, the BERT encoder takes the attention weights calculated based on the phrase embeddings and produces a limited output 1760.

[0087] Figure 18 shows an exemplary training method 1800 for a tag-based converter. The training method 1800 includes training text 1810, GUESS tags 1820, a tag-based converter 1830, and a word error rate calculator 2840. Furthermore, the training method has a BERT model 1850 and a cross-entropy calculator 1860. Operationally, training begins with the training text 1810 being fed to GUESS tags 1820, which infer the types of text present in the input text 2810. GUESS tags 1820 first sorts the tags of each character by character tag frequency. This operation generates easily obtainable weak surveillance information. Next, consecutive digits share the same tags. Finally, the computational cost is estimated in advance, and if the predicted time cost is high, the possible tags are limited to the top 5.

[0088] The inferred tags are applied to the input text and then passed to both the tag-based converter 1830 and the BERT model 1850. For convenience, we will first describe the tag-based converter 430 and the word error rate 440. The tag-based converter 1830 converts the input text string into text based on tags from the GUESS tags 420. For example, the tag-based converter 1830 can convert numbers and symbols into text. Next, after conversion to text, the string is passed to a word error rate checker (WER) which checks the amount of errors generated after the conversion. If the number of errors is high, the conversion is sent back through the process with different tags to retrain the method to reduce the number of errors. If the WER is low, take training takes only sentences with a WER of 0 and uses them to train the BERT model 1850. The BERT model 1850 is then applied to assist in label inference, and a second BERT model 18450 is trained on the relabeled corpus. The output of the BERT model 1850 is obtained, and the cross-entropy 1860 is calculated.

[0089] Figure 19 shows in detail one embodiment of the text normalization device 1900. The text normalization device 1900 includes input text 1910, which is fed into bidirectional encoder representations from character-based transformers (BERT) 1920. The output of BERT 1920 is then acted upon by conditional random fields (CFR) 1930. After processing with CFR 1930, the normalization device 1900 finally outputs normalized text 1940.

[0090] Input text 1910 includes a string of text consisting of combinations of characters, including text and other markers such as punctuation. For example, input text 1910 may be a string such as a date. A date typically includes a mixture of markers and text. There are various forms that contain the same information, such as "February 2, 2014" or "2 / 2 / 14". Other examples may be a text message that typically includes other markers such as emojis, or an email that includes various punctuation marks such as the "@" symbol. Another example of input text 1910 may be a metric or measurement, which may be ambiguous and without context. For example, "16M" may mean "16 megabytes" or "16 meters" depending on how it is read. Finally, as a non-exhaustive list, input text 1910 may be mathematics, a website URL, program code, language text, a combination of languages, any combination of the examples above, or any unique marker.

[0091] After receiving the input text 1910, the text normalization machine 1900 passes the text to the character-based BERT 1520. The character-based BERT 1920 is fully trained and labels each character with predefined tags. In some embodiments, BERT 1920 is a full-size BERT, and in other embodiments, it is a single-layer BERT model. Single-layer BERTs are generally faster than full-size BERTs, but smaller modules can result in performance degradations such as a higher error rate. After tagging the input text, BERT 1920 passes that information to the rule CFR 1930. The Conditional Random Field (CFR) loss function labels entire segments instead of individual characters. CFR 1930 receives the tagged text, parses the text, and finally produces the normalized text output 2940.

[0092] The experimental results and embodiments of the disclosure provided herein offer technical improvements that enable the extraction of NMs from unstructured corpora. In the embodiments, a first NM extraction dataset is annotated for novel-based extraction, and three baselines are developed to extract single-span or multi-span NMs of target utterances from the surrounding context. Furthermore, the embodiments use the extractor to extract 749K (context, utterance, NM) triples from the novel and investigate whether its use improves NM generation through semi-supervised learning. The experimental results demonstrate that automatically extracted triples serve as high-quality augmentation data of clean triples extracted from scripts to generate more relevant, fluent, valid, and factually consistent NMs than a purely supervised generator, and the resulting generator can assist dialogue comprehension tasks such as dialogue machine reading and sentiment classification by simply adding predicted "unspoken" NMs to each utterance or story in the input.

[0093] For example, instead of scripts, novels contain a wealth of nonverbal messages (NM) in the author's own words, in addition to what the characters say, and thousands of novels have already been adapted into scripts (primarily by professional scriptwriters). Furthermore, experimental results reported herein have observed that the density of NM in novels is higher than that of scripts based on annotated corpora (approximately 67.4%), indicating the potential to leverage novels for NM extraction. Thus, embodiments demonstrate using this unstructured resource to mitigate the problem of insufficient NM data that hinders the full utilization of deep neural models. Since this direction was not unexplored, embodiments first define the task as extracting one or more spans from the context surrounding a target utterance and annotate a first nonverbal message extraction dataset, the NME, based on three novels containing 4K (context, utterance, NM) instances. Furthermore, three baselines (pattern, extract, and generate) are used to extract NM and evaluate them in the NME.

[0094] Embodiments of this specification address the question of whether a computer can automatically build data to improve NM generation using a novel, unlabeled corpus. For example, an embodiment uses a trained extractor to extract 749K pseudo-labeled instances from hundreds of novels and trains a generator based on different backbone models to generate nonverbal messages given a single target utterance and its context. Experiments show that these triples serve as high-quality augmentation data of clean triples extracted from well-structured scripts, producing more relevant, fluent, valid, and virtually consistent NMs. Furthermore, the semi-supervised generator according to the embodiment demonstrates its usefulness by assisting NM-less Chinese dialogue and narrative comprehension tasks such as the dialogue subset of the machine reading dataset C3 and the sentiment classification EWECT, by simply adding the generated "unspoken" NMs to each input utterance or narrative.

[0095] Therefore, embodiments of this specification provide the design and annotation of a first NM extraction dataset based on an unstructured corpus, several robust nonverbal message extraction and generation baselines on different backbone models, extraction of large amounts of (context, utterance, NM) data from an unlabeled unstructured corpus using an NM extractor, and demonstration of the usefulness of the data for improving the performance of NM generation. The experimental results presented herein demonstrate that the NM generator can assist in dialogue comprehension tasks in computer technology.

[0096] The techniques described above can be implemented as computer software using computer-readable instructions and physically stored on one or more computer-readable media or by one or more specially configured hardware processors. For example, Figure 20 shows a computer system 2000 suitable for implementing a particular embodiment of the subject matter of this disclosure.

[0097] Computer software can be coded using any suitable machine code or computer language that can be processed by mechanisms such as assembly, compilation, and linking to generate code containing instructions that can be executed directly or through interpretation, microcode execution, etc., by a computer's central processing unit (CPU), graphics processing unit (GPU), etc.

[0098] The instructions can be executed on various computers or their components, including, for example, personal computers, tablet computers, servers, smartphones, game consoles, Internet of Things devices, etc.

[0099] The components shown in Figure 12 of Computer System 2000 are illustrative and do not imply any limitation on the scope of use or functionality of computer software implementing embodiments of this disclosure. Furthermore, the configuration of the components should not be construed as having any dependencies or requirements related to any one or combination of the components shown in the exemplary embodiments of Computer System 2000.

[0100] The computer system 2000 may include certain human interface input devices. Such human interface input devices may respond to input from one or more human users through, for example, sensory input (e.g., keystrokes, swipes, data glove movements), voice input (e.g., voice, clapping), visual input (e.g., gestures), and olfactory input (not shown). The human interface devices may also be used to capture certain media that do not necessarily need to be directly related to conscious human input, such as voice (e.g., conversation, music, ambient sounds), images (e.g., scanned images, photographic images taken from a digital camera), and video (e.g., including 2D video, 3D video, and stereoscopic video).

[0101] The input human interface device may include one or more of the following (only one is shown): keyboard 2001, mouse 2002, trackpad 2003, touchscreen 2010, joystick 2005, microphone 2006, scanner 2008, and camera 2007.

[0102] The computer system 2000 may also include certain human interface output devices. Such human interface output devices may stimulate the senses of one or more human users, for example, through sensory output, sound, light, and smell / taste. Such human interface output devices may include sensory output devices (e.g., touchscreen 2010, or joystick 2005 (for sensory feedback, however, there may also be sensory feedback devices that do not function as input devices)), sound output devices (e.g., speaker 2009, headphones (not shown)), and visual output devices (e.g., screen 2010, CRT screen, LCD screen, plasma screen, OLED screen, each having or not having touchscreen input capability, each having or not having sensory feedback capability, some of which may be capable of outputting two-dimensional visual output or three-dimensional or more output through means such as stereoscopic output, virtual reality glasses (not shown), holographic displays, and smoke tanks (not shown), and printers (not shown))).

[0103] Computer system 2000 may also include human-accessible storage devices, as well as related media such as optical media like CD / DVDROM / RW2020 with media such as CD / DVD2011, thumb drives 2022, removable hard drives or solid-state drives 2023, legacy magnetic media such as tapes and floppy disks (not shown), and devices based on dedicated ROM / ASIC / PLD such as security dongles (not shown).

[0104] Those skilled in the art should also understand that the term “computer-readable medium” as used in connection with the subject matter of this disclosure does not include a transmission medium, carrier wave, or other transient signal.

[0105] The computer system 2000 may also include an interface 2099 to one or more communication networks 2098. Networks 2098 may be, for example, wireless, wired, or optical. Networks 2098 may further be local, wide-area, urban, vehicle and industrial, real-time, latency-tolerant, etc. Examples of networks 2098 include local area networks such as Ethernet, cellular networks including wireless LAN, GSM, 3G, 4G, 5G, LTE, etc., wired or wireless wide-area digital networks including cable TV, satellite TV, and terrestrial broadcast TV, vehicle and industrial networks including CANBus, etc. Certain networks 2098 generally require specific general-purpose data ports or peripheral buses (2050 and 2051) (for example, an external network interface connected to a USB port on the computer system 2000). Others are generally integrated into the core of the computer system 2000 by being connected to a system bus as described below (for example, an Ethernet interface to a PC computer system, or a cellular network interface to a smartphone computer system). Using these networks 2098, the computer system 2000 can communicate with other entities. Such communication can be unidirectional reception only (e.g., broadcast TV), unidirectional transmission only (e.g., CANbus to a specific CANbus device), or bidirectional to other computer systems using, for example, a local or wide-area digital network. Specific protocols and protocol stacks may be used for each of the networks and network interfaces described above.

[0106] The aforementioned human interface device, human-accessible storage device, and network interface can be mounted on the core 2040 of the computer system 2000.

[0107] The core 2040 may include one or more central processing units (CPUs) 2041, graphics processing units (GPUs) 2042, graphics adapters 2017, dedicated programmable processing units in the form of FPGAs 2043, hardware accelerators 2044 for specific tasks, etc. These devices may be connected via the system bus 2048, along with read-only memory (ROM) 2045, random access memory 2046, and internal mass storage devices 2047 such as internal, user-inaccessible hard drives, SSDs, etc. In some computer systems, the system bus 2048 is accessible in the form of one or more physical plugs to allow expansion with additional CPUs, GPUs, etc. Peripherals can be attached directly to the core's system bus 2048 or via the peripheral bus 2049. The architecture of the peripheral bus includes PCI, USB, etc.

[0108] The CPU 2041, GPU 2042, FPGA 2043, and accelerator 2044 can execute specific instructions that, when combined, can generate the aforementioned computer code. This computer code can be stored in ROM 2045 or RAM 2046. Temporary data can also be stored in RAM 2046, while permanent data can be stored, for example, in the built-in mass storage device 2047. High-speed storage and retrieval to any of the memory devices can be enabled through the use of cache memory that may be closely associated with one or more of the CPU 2041, GPU 2042, mass storage device 2047, ROM 2045, RAM 2046, etc.

[0109] Computer-readable media may contain computer code for performing actions performed by various computers. The media and computer code may be specifically designed and configured for the purposes of this disclosure, or they may be of a type well known and available to those skilled in the computer software field.

[0110] As an example, and not limited to, a computer system 2000 having an architecture, and specifically the core 2040, can provide functionality as a result of a processor (including a CPU, GPU, FPGA, accelerator, etc.) executing software embodied in one or more tangible computer-readable media. Such computer-readable media may be specific storage devices of the core 2040 having non-transient characteristics, such as the core-integrated mass storage device 2047 or ROM 2045, and media associated with the user-accessible mass storage devices described above. Software implementing various embodiments of this disclosure can be stored in such devices and executed by the core 2040. The computer-readable media may include one or more memory devices or chips, depending on the specific needs. The software can cause the core 2040 and specifically the processors within it (including a CPU, GPU, FPGA, etc.) to execute specific processes or specific parts of specific processes described herein, including defining and modifying data structures stored in RAM 2046 according to software-defined processes. As an addition or alternative, a computer system may provide functionality as a result of a logic hardwired or other circuit implementation (e.g., accelerator 2044) that can operate together with or in place of the software to perform the specific processes or specific parts of the specific processes described herein. References to software include logic, and vice versa, where appropriate. References to computer-readable media may include, where appropriate, circuits (such as integrated circuits (ICs)) that house software for execution, circuits that implement logic for execution, or both. This disclosure includes any appropriate combination of hardware and software.

[0111] While this disclosure describes several exemplary embodiments, alternatives, substitutions, and various equivalents exist and are included within the scope of this disclosure. As will be apparent to those skilled in the art, numerous systems and methods can be devised to implement the principles of this disclosure and thus fall within the spirit and scope of this disclosure, although these are not expressly shown or described herein.

Claims

1. A method which is executed by at least one processor, The steps include receiving text containing multiple sentences, A machine learning model extracts a nonverbal message from one of the sentences and adds an annotation to the text, wherein the annotation indicates the nonverbal message. The steps include: outputting a version of the text that includes the aforementioned annotations; A method that includes this.

2. The method according to claim 1, wherein the extraction of the nonverbal message comprises determining the type of the nonverbal message, the type being at least one of motion type, internal state type, pause type, and voice type.

3. The aforementioned movement type includes at least one physical movement other than facial expression and one facial expression. The aforementioned internal state type indicates at least one of intention and emotion, The aforementioned pause type indicates the passage of time, The method according to claim 2, wherein the voice type indicates at least one of listener, number of speakers, tone, volume, speed, pitch, and timbre.

4. The method according to claim 1, wherein extracting the nonverbal messages includes determining the context of a sentence based on determining a predetermined number of sentence contexts both before and after the sentence using a machine learning model.

5. The extraction of the aforementioned nonverbal message is based on minimizing the following equation: [Math 1] Here, V represents a set of nonverbal message extraction instances, θ represents the parameters of the extraction model of the machine learning model, and L is defined by the following equation: [Math 2] t represents an instance of the aforementioned sentence, and p start and p end This is a vector representing the estimated probability of tokens in response span a, and a start and a end The method according to claim 4, wherein each represents the start offset and end offset of the answer span a.

6. The aforementioned machine learning model is trained based on minimizing the following damage objective function. [Math 3] Here, y t x represents the nonverbal message of the above sentence, t θ represents the context within a predetermined distance of the aforementioned sentence, e and θ d The method according to claim 5, wherein is the parameter of the machine learning model.

7. The above annotation is based on the following probability determination: [Math 4] The method according to claim 1, wherein U represents the utterance of the sentence, and n represents a nonverbal message including the nonverbal message.

8. The method according to claim 1, wherein the step of receiving the text includes the step of inputting at least one of a movie script and a novel into the machine learning model.

9. The method according to claim 1, wherein the first sentence indicates that the speaker uttered a conversation, and the annotation indicates that the conversation was uttered as a shout.

10. The method according to claim 1, wherein the text indicates multiple speakers.

11. It is a device, At least one memory configured to store computer program code, At least one processor, Includes, The computer program code, when executed by the at least one processor, causes the device to perform the method according to any one of claims 1 to 10.

12. A computer program that causes a computer to perform the method described in any one of claims 1 to 10.

Images