Multilingual Grammar Error Correction

Abstract

A method (400) for training a text generation model (122) for grammatical error correction (GEC) includes obtaining a multilingual set (210) of text samples (212). Each text sample includes a monolingual textual representation of a respective sentence. The operations also include, for each text sample of the multilingual set of text samples, generating a modified synthetic version (222) of the respective text sample, the modified synthetic version of the respective text sample including grammatical changes to the monolingual textual representation of the respective sentence associated with the respective text sample. The operations further include training the text generation model using a training set (230) of sample pairs (232). Each sample pair includes one of the respective text samples of the multilingual set of text samples and a corresponding modified synthetic version of the one of the respective text samples of the multilingual set of text samples.

Description

[Technical Field]

[0001] The present disclosure relates to multilingual grammar error correction. [Background technology]

[0002] As user-generated text continues to play a key role in human-computer and human-human interactions using computing devices, the ability of natural language generation (NLG) systems to ensure that user-generated text is grammatically accurate can be an important aspect of communication. For example, grammatically accurate text can enable readability and avoid potential miscommunication or misunderstanding. That is, grammatical errors can change the meaning of a communication or lead to some degree of confusion regarding the meaning of the text. While traditional grammatical error correction techniques attempt to address grammatical issues in text, such techniques often suffer from issues with training data (e.g., insufficient training data, label accuracy of the training data, and / or lack of bias in the error distribution for the training data), rendering grammatical error correction models of limited capability. Summary of the Invention [Means for solving the problem]

[0003] One aspect of the present disclosure provides a computer-implemented method for training a text generation model for grammatical error correction (GEC). The method, when executed by data processing hardware, causes the data processing hardware to perform operations. The operations include obtaining a multilingual set of text samples, each text sample including a monolingual text representation of a respective sentence. The operations also include, for each text sample in the multilingual set of text samples, generating a modified synthetic version of the respective text sample, the modified synthetic version of the respective text sample including grammatical changes to the monolingual text representation of the respective sentence associated with the respective text sample. The operations further include training the text generation model using a training set of sample pairs. Each sample pair in the training set of sample pairs includes one of the respective text samples in the multilingual set of text samples and a corresponding modified synthetic version of the one of the respective text samples in the multilingual set of text samples.

[0004] Another aspect of the present disclosure provides a system for training a text generation model for grammatical error correction (GEC). The system includes data processing hardware and memory hardware in communication with the data processing hardware. The memory hardware stores instructions that, when executed on the data processing hardware, cause the data processing hardware to perform operations. The operations include obtaining a multilingual set of text samples, each text sample including a monolingual text representation of a respective sentence. The operations also include, for each text sample in the multilingual set of text samples, generating a modified synthetic version of the respective text sample, the modified synthetic version of the respective text sample including grammatical changes to the monolingual text representation of the respective sentence associated with the respective text sample. The operations further include training the text generation model using a training set of sample pairs. Each sample pair in the training set of sample pairs includes one of the respective text samples in the multilingual set of text samples and a corresponding modified synthetic version of the one of the respective text samples in the multilingual set of text samples.

[0005] Implementations of methods or systems of the present disclosure may include one or more of the following features. In some implementations, the operations further include, after training the text generation model, fine-tuning the trained text generation model using supervised training data, the supervised training data including non-synthetic text pairs, each non-synthetic text pair including a grammatically incorrect text sample and a grammatically correct text version of the grammatically incorrect text sample. In some examples, generating the modified synthetic version of each text sample includes removing two or more characters from each sentence associated with the respective text sample. In some configurations, generating the modified synthetic version of each text sample includes replacing a first set of characters from each sentence associated with the respective text sample with a second set of characters different from the first set of characters. In some implementations, generating the modified synthetic version of each text sample includes inserting one or more characters into each sentence associated with the respective text sample. In some cases, generating the modified synthetic version of each text sample includes changing the character case of characters in words of each sentence associated with the respective text sample. The text generation model may include a transformer encoder-decoder architecture. The operations may further include pre-training the text generation model using a multilingual training corpus based on a masked-language objective.

[0006] These implementations of the disclosed methods or systems may also include generating an altered composite version of each text sample by randomly applying an alteration operation to each sentence associated with the respective text sample, where each altered composite version is unique with respect to other altered composite versions of the text sample. The alteration operation may include at least one of removing two or more characters from each sentence associated with the respective text sample, replacing a first set of characters from each sentence associated with the respective text sample with a second set of characters different from the first set of characters, inserting one or more characters into each sentence associated with the respective text sample, or changing the case of characters of words in each sentence associated with the respective text sample. The operations of the present disclosure may also include using the trained text generation model for GEC during inference by (i) receiving, as input to the trained text generation model, a first input text in a first language that includes grammatical errors, and generating, as output from the trained text generation model, a first output text in the first language that corrects the grammatical errors; and (ii) receiving, as input to the trained text generation model, a second input text in a different second language that includes grammatical errors, and generating, as output from the trained text generation model, a second output text in the second language that corrects the grammatical errors.

[0007] The details of one or more implementations of the disclosure are set forth in the accompanying drawings and the description below. Other aspects, features, and advantages will be apparent from the description and drawings, and from the claims. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a schematic diagram of an exemplary text correction environment using a grammar error correction model. [Figure 2A] FIG. 1 is a schematic diagram of the training process for a grammar error correction model. [Figure 2B]FIG. 1 is a schematic diagram of the training process for a grammar error correction model. [Figure 2C] FIG. 1 is a schematic diagram of the training process for a grammar error correction model. [Figure 3] FIG. 1 is a schematic diagram of the fine-tuning process for a grammar error correction model. [Figure 4] 1 is a flowchart of an example sequence of operations for a method of training a text generation model for grammar error correction. [Figure 5] FIG. 1 is a schematic diagram of an example computing device that can be used to implement the systems and methods described herein. DETAILED DESCRIPTION OF THE INVENTION

[0009] Like reference symbols in the various drawings indicate like elements.

[0010] Natural language processing (NLP) generally refers to the use of computing technology to understand natural human language. NLP enables user devices or computing devices to automatically process forms of natural human language, such as speech or text. Another specific type of natural language processing is called natural language generation (NLG). NLG broadly relates to the process of generating textual representations of human language (i.e., text). NLG can include a wide range of text generation tasks, such as grammatical error correction (GEC). As text becomes increasingly prevalent in the use of computing devices, there is a greater demand for GEC, which refers to the task of correcting grammatical errors or other text-related errors. For example, in an era where text messaging has become more widely used than phone calls, GEC may improve the readability of user-generated text. Furthermore, improved GEC may also assist people who face grammatical challenges, such as non-native speakers, children, or individuals with some types of disabilities.

[0011] Traditionally, the task of GEC has been considered monolingual text-to-text transcription. To perform text-to-text transcription, text generation models often employ neural network architectures. These neural network architectures generally require fairly large training datasets. Furthermore, the need for fairly large training datasets can be problematic across multiple languages. For example, while widespread languages ​​such as English or Japanese have a large amount of training data, other languages ​​suffer from a lack of training data. Less widespread languages ​​tend to have an inadequate amount of training data to train neural network text generation models capable of performing GEC. Thus, while there are training datasets that include multiple languages, even these training datasets tend to be highly biased, with the distribution of training samples in these datasets favoring more widespread languages ​​and underrepresenting other languages. For example, one common training corpus includes over 80 languages, but only 10 of these languages ​​contain more than 10,000 training samples (e.g., 10,000 grammatically incorrect-grammatically correct sample pairs). To put this in perspective, the most popular languages ​​in this general training corpus, such as English and Japanese, have over a million training example pairs. Given this inherent language bias, any text generation model trained for GEC using this training data may perform suboptimally for many languages ​​(e.g., less popular languages).

[0012] Due to the limited amount of suitable training data for performing GEC, there have been efforts to supplement or use synthetic training data. Synthetic data refers to data that is machine-generated (e.g., from a machine model) rather than human-generated. Unfortunately, models taught by synthetic training data are not without drawbacks. For example, fine-tuning models for GEC tasks has been shown to improve GEC accuracy, but it often requires language-specific tuning (e.g., with language-specific hyperparameters and spelling dictionaries) and / or has difficulty representing the complete error distribution for the training evaluation set. Due to these issues, the final model for GEC typically undergoes a multi-stage fine-tuning process, requiring a specific learning rate and training steps for each fine-tuning stage.

[0013] To address some of these issues with teaching a model to perform GEC, implementations herein are directed to a pre-training approach that applies a fully unsupervised, language-agnostic pre-training objective that mimics the corrections typically found in labeled data. The pre-training approach is unsupervised because the training data used for pre-training comes from grammatically correct text samples (e.g., grammatically correct sentences) paired with synthetic, ungrammatically incorrect text versions of themselves. This means that the training sample pairs for the pre-training process include a grammatically correct text sample along with a version of itself that has been automatically modified (i.e., grammatically altered to be ungrammatical by a modification operation). Here, because a machine generates this modified version, the modified version is synthetic and not human-created. For context, it is true that conventionally, an ungrammatical text sample is paired with a label that is a grammatically correct version of the ungrammatical text sample. Instead, by synthetically altering originally grammatically correct text samples, training sample pairs do not require explicit labels (e.g., labels identifying grammatical corrections).

[0014] The process of generating synthetic training text samples for training sample pairs based on grammatically correct text samples is also language-agnostic. Here, the modification techniques are not language-specific, so the goal is language-agnostic. That is, the techniques modify each grammatically correct text sample without focusing on the language on which the grammatically correct text sample is based. For example, changing a sequence of characters or tokens in a grammatically correct sample does not introduce any bias toward a specific language. Additionally, the techniques also aim, in some respects, to prevent bias against any specific types of grammatical errors that may be specific to a certain language. For example, if a modification operation changes an "e" before an "i," which is a common grammatical error in English, a model performing GEC may become inherently biased and learn to identify English grammatical errors rather than multilingual errors more broadly. By using modification operations and techniques that are not specific to a particular language, the modification process may prevent teaching the text generation model any form of language correction bias. Furthermore, unlike previous approaches that may generate synthetic training data, the synthetic pre-training process remains completely language-agnostic by training a single model for all languages ​​in the training dataset without employing language-specific priors.

[0015] In some examples, after pre-training a text generation model for GEC, the text generation model is considered a GEC model. Here, the GEC model may undergo a fine-tuning process prior to inference. In this fine-tuning process, the GEC model receives supervised GEC training data. In some implementations, the GEC training data is language-specific. In other implementations, the GEC training data corresponds to non-synthetic or human-created text samples that are available with appropriate labels. That is, the GEC training data are human-created text samples, and each training example is a grammatically incorrect-grammatically correct sample pair.

[0016] FIG. 1 illustrates an example of a text correction environment 100. In the text correction environment 100, a user 10 operates a user device 110 to engage in a text conversation with another person. For example, FIG. 1 illustrates the user device 110 as a mobile phone, and the user 10 inputs text into a keyboard (e.g., a touch-input keyboard). The user device 110 (also generally referred to as device 110) may correspond to any computing device associated with the user 10 and capable of generating text 126. Some examples of the user device 110 include, but are not limited to, mobile devices (e.g., mobile phones, tablets, laptops, etc.), computers, wearable devices (e.g., smart watches), smart appliances, Internet of Things (IoT) devices, smart speakers, etc. The user device 110 includes data processing hardware 112 and memory hardware 114 in communication with the data processing hardware 112 that stores instructions that, when executed by the data processing hardware 112, cause the data processing hardware 112 to perform one or more operations (e.g., operations related to text generation or text correction). The user device 110 may also include an audio subsystem 116 comprising an audio capture device (e.g., microphone) 116, 116a for capturing and converting spoken words into electrical signals in the voice-enabled system 100, and an audio output device (e.g., speaker) 116, 116b for communicating an audible audio signal (e.g., as output audio data from the device 110). The user device 110 (e.g., using the hardware 112, 114) is further configured to perform grammatical error correction (GEC) on a text sequence 126 input by the user 10 using a GEC model 124 of the NLG system 120.

[0017] The NLG system 120 refers to a natural language generation system capable of processing text (e.g., user-generated text 126) for various functions or tasks. Here, the NLG system 120 includes a text generation model 122. The text generation model 122 is a model with a flexible NLP framework that can be further trained for specific language tasks. In some examples, the text generation model 122 is a model taught by transfer learning. For example, the text generation model 122 is pre-trained on available unlabeled text data using a self-supervised task (i.e., a data-rich task). In some implementations, the text generation model 122 is a transformer encoder-decoder model that can be further fine-tuned for many specific NLG tasks. More specifically, the text generation model 122 may be a text-to-text transfer transformer (T5) with an integrated framework that converts text-based language problems into a text-to-text format. By using a text-to-text framework, the text generation model 122, along with its loss function and hyperparameters, can be adapted to many (if not all) NLP tasks, such as machine translation, document summarization, question answering, classification tasks, GEC tasks, etc. Furthermore, when the text generation model 122 is pre-trained, the pre-training corpus may include multiple languages, allowing downstream task-specific versions of the text generation model 122 to potentially be multilingual models as well. For example, one common pre-training corpus includes over 100 languages.

[0018] In some examples, the pre-training process for the text generation model 122 is based on some version of a masked language objective (e.g., a span prediction task). After pre-training the text generation model 122, the text generation model 122 is further trained (e.g., by the training process 200 and the fine-tuning process 300) to be capable of performing GEC. Now, once the text generation model 122 has this GEC capability, the text generation model 122 is referred to as a GEC model 124. In other words, the GEC model 124 is a downstream version of the text generation model 122 for performing the NLG task of grammar error correction. That is, although the text generation model 122 is a fairly omnipotent model from the perspective of the NLP task, the text generation model 122 is generally not accurate for a specific NLG task such as GEC until it undergoes further task-specific training. This means that the text generation model 122 is first pre-trained to be an omnipotent NLP model and then trained (e.g., by the training process 200) to be a GEC model 124. As will be explained in more detail below, the GEC model 124 may therefore perform GEC to some extent without this further GEC refinement, but may be further refined (eg, by refinement process 300) for greater GEC accuracy.

[0019] In some implementations, device 110 communicates with remote system 140 via network 130. Remote system 140 may include remote resources 142, such as remote data processing hardware 144 (e.g., a remote server or CPU) and / or remote memory hardware 146 (e.g., a remote database or other storage hardware). Device 110 may utilize remote resources 142 to perform various functions related to text generation and / or GEC. For example, some of the functionality of NLG system 120 may reside on remote system 140. In one example, NLG system 120 may reside on device 110 to perform on-device text generation (e.g., GEC). In another example, NLG system 120 resides on a remote system to provide server-side GEC. In yet another example, the functionality of NLG system 120 is split across device 110 and server 140.

[0020] In some examples, such as FIG. 1 , a user 10 interacts with a program or application 118 on a user device 110 that uses an NLG system 120, and more specifically, a GEC model 124. Here, the application 118 corresponds to a communication application, such as a short message service (SMS) application or a multimedia messaging service (MMS) application. The user 10 may explicitly invoke the application 118 (e.g., by user input) to generate text 126. For example, FIG. 1 shows a user 10, Ted, communicating (e.g., sending text messages) with another user, a friend of Ted's named Jane. Here, a text message conversation between Ted 10 and Jane may be received / generated by the communication application 118 on the user device 110. In these examples, the communication application 118 uses an NLG system 120 with a text generation model 122 that is further trained to be a GEC model 124 for the specific task of GEC. Here, user-generated text 126 is sent as input to GEC model 124 (i.e., text generation model 122 trained for GEC). In this example, user-generated text 126 is in English and contains two or more grammatical errors. GEC model 124 receives text 126 and generates as output (i.e., an output text sequence) a sequence of text (referred to as grammatically correct text 128) that corrects the grammatical errors contained in user-generated text 126. In the illustrated example, user 10 receives an initial text message from Jane asking, "What time do you think you will leave?" User 10 responds to this question by typing text 126: "Well its 3 pm now, and I want to run a quick errund on the way their. I'm thinking 4:15." Here, user 10's text 126 does not capitalize the first letter of the first word of text 126, "well."Text 126 also misspells the word "quick" as "quicke" and the word "errand" as "errund." Text 126 shows that the user mistakenly used "me" instead of "I" when referring to the first person. Text 126 also shows that user 10 misused the possessive word "their" when he intended to type the adverb "there." User 10 also mistakenly used the possessive "its" when he intended the conjunction "it's," and did not include an apostrophe in the conjunction "I'm." GEC model 124 recognizes these grammatical errors along with text 126 and generates as output a corrected form 128 of text 126, such that grammatically corrected text 128 can be sent as a text message to Jane. In this example, the output of GEC model 124, corrected text 128, presents "Well, it's 3 pm now. And I want to run a quick errand on the way there. I'm thinking 4:15 pm."

[0021] While the illustrated example shows English text 126, GEC model 124 may correct grammatical errors in multiple languages. Continuing with this example, this means that user 10, Ted, may later have a text conversation in Spanish with his friend Steve, and the same GEC model 124 may correct Spanish grammatical errors in the text 126 entered by Ted. In other words, GEC model 124 may generate first output text 128 to correct grammatical errors in first text 126 in a first language and also generate second output text 128 to correct grammatical errors in second text 126 in a second language different from the first language. As a multilingual GEC model 124, GEC model 124 may grammatically correct errors in text in several languages ​​(e.g., two languages, ten languages, eighty languages, or more than one hundred languages).

[0022] 2A-2C illustrate a training process 200 for teaching a texting-generating model 122 to perform GEC. Once the training process 200 teaches the text generation model 122 to perform GEC, the text generation model 122 becomes a GEC model 124. In other words, without specific GEC training, the text generation model 122 cannot perform the text generation task of GEC. The training process 200 obtains a corpus 210 of text samples 212 that is used to train the text generation model 122 to perform GEC. The corpus 210 generally refers to a collection or multilingual set of text samples 212. The corpus 210 is a multilingual set of text samples 212 in that each individual text sample 212 is monolingual (e.g., a monolingual sentence), but the corpus 210 includes text samples 212 from two or more languages. For example, the corpus 210 includes text samples 212 from dozens or even more than a hundred different languages.

[0023] In conjunction with the text sample 212, the training process 200 uses a corrupter 220 to generate an altered synthetic version 222 of the text sample 212 (also referred to as an altered text sample 222 or altered text 222). That is, the corrupter 220 is configured to generate a machine-generated version of the text sample 212, making the altered text version 222 a synthetic text sample. Generally, the text sample 212 that the corrupter 220 alters is a non-synthetic text sample derived from human-generated text. In other words, the corrupter 220 functions to "corrupt," i.e., introduce grammatical errors into the text sample 212. In this regard, the text sample 212 serves as a grammatically correct text sample that the corrupter 220 grammatically modifies to produce the altered synthetic version 222 of the text sample 212. For example, the corruptor 220 corrupts the text sample 212 to form a modified synthetic text version 222 of the text sample 212 by making grammatical changes to the monolingual text representation of the text sample 212. The training process 200 then pairs the text sample 212 with its modified synthetic text version 222 to form training sample pairs 232. That is, the text sample 212 and the modified synthetic version 222 form grammatically correct-ungrammatically correct text sample pairs 232. The training process 200 compiles the collection of training sample pairs 232 into a training set 230 that will be used to train the text generation model 122 to perform GEC (i.e., into the GEC model 124). When the training process 200 trains the text generation model 122 using the training set 230, the training process 200 may train the text generation model 122 until convergence (i.e., when the model 122 outputs corrected text 128 for the GEC that converges or matches the provided text samples 212).In Figures 2A-2C, "original" or "O" refers to the text sample 212 from the corpus 210 in the training sample pair 232, and "modified" or "C" refers to the modified synthetic text version 222 of the text sample 212 (generated by the corruptor 220) in the training sample pair 232.

[0024] 2B , the corruptor 220 may be configured to perform various alteration operations that change portions of the text sample 212. For example, FIG. 2B illustrates that the corruptor 220 may perform alteration operations that remove portions of the text sample 212, replace portions of the text sample 212, insert portions of the text sample 212, or change the word / sentence structure (e.g., punctuation and / or capitalization) of the text sample 212. An exemplary deletion operation may be deleting one or more characters or several spans of tokens. In FIG. 2B , the first altered text sample 222, 222a, shows that the character "ro" in "grouping" has been deleted for the text sample 212, "A paragraph is a grouping of sentences." FIG. 2B also illustrates an example of a substitution alteration operation in which one or more characters may be replaced or exchanged with one or more other characters. In this example, the second altered text sample 222, 222b replaces the letter "e" in the word "sentences" with the letter "a" for the text sample 212, "A paragraph is a grouping of sentences." Similarly, the collapsor 220 replaces the token "ph" in "paragraph" with "fe" to form the third altered text sample 222, 222c. The collapsor 220 performs a collapsing operation to insert one or more characters to form the fourth altered text sample 222, 222d by inserting an extra "p" in the word "grouping." The fifth altered text sample 222, 222e is an example in which the collapsor 220 changes the case of letters in one or more words of the text sample 212. Here, in the fifth altered text sample 222, 222e, the corruptor 220 has changed the initial "A" to a lowercase "a" and capitalized the "P" in "paragraph."Corruptor 220 may perform one or more modification operations when generating modified text samples 222. Additionally or alternatively, the modification operations are not an exhaustive list of text sample changes that corruptor 220 may perform to introduce grammatical errors. In some configurations, when generating training set 230 of training sample pairs 232, corruptor 220 may randomly apply various modification operations to ensure variations of grammatical errors within training set 230. For example, each modified text sample 222 may be unique with respect to other modified text samples 222 in training set 230.

[0025] In some implementations, such as FIG. 2C , the training process 200 also includes a segmenter 240. The segmenter 240 is configured to receive the text sample 212 and generate segmented samples 242 from the text sample 212. A segmented sample 242 refers to a length or portion of the text sample 212. That is, the segmented sample 242 may be a specific amount of words or sentences from the text sample 212. In this regard, the segmenter 240 may function to ensure that the size of the altered text sample 212 is compatible with the corruptor 220 and, more generally, an acceptable size for training the text generation model 122. For example, the text sample 212 may correspond to a paragraph of text, and the segmenter 240 converts the text sample 212 into sentences such that each segmented sample 242 represents a sentence of the text sample 212. In FIG. 2C , the segmenter 240 receives the text sample 212, which is a paragraph of three sentences. Using this text sample 212, the segmenter 240 generates three segmented text samples 242, 242a-c, where the first segmented text sample 242a is the first of the three sentences, the second segmented text sample 242b is the second of the three sentences, and the third segmented text sample 242c is the third of the three sentences.

[0026] 3, after the training process 200 trains the text generation model 122 to become the GEC model 124 using the training dataset 230, the GEC model 124 may be further trained in a fine-tuning process 300. During the fine-tuning process 300, the fine-tuning process 300 may use supervised training data. In some examples, the supervised training data corresponds to a set of non-synthetic text pairs 312, 312a-n, that form a fine-tuned training corpus 310. Each fine-tuned text pair 312 includes a grammatically correct text sample 314 (e.g., designated "G" in FIG. 3) and an ungrammatically incorrect text sample 316 (e.g., designated "UG" in FIG. 3). Here, the ungrammatical text sample 316 is not generated by the corruptor 220, but is human-generated text. The grammatically correct text samples 314 refer to grammatically correct versions of the ungrammatically correct text samples 316, which are human-generated text, such that the grammatically correct text samples 314 serve as labeled fine-tuning training data or supervised fine-tuning training data. The fine-tuning training corpus 310 may be a multilingual corpus similar to the corpus 210 used during the training process 200. That is, each text pair 312 in the fine-tuning training corpus 310 corresponds to a multilingual grammatical-ungrammatical pair, but the fine-tuning training corpus 310 has pairs 312 in different languages. In some examples, the GEC model 124 is trained by the fine-tuning process 300 using the fine-tuning training corpus 310 until the exact match accuracy of the GEC model 124 on the development dataset decreases (e.g., after 200 steps, 800k examples, or 7 epochs).

[0027] 4 is a flowchart of an example sequence of operations for a method 400 of training a text generation model 122 for grammatical error correction. At operation 402, the method 400 obtains a multilingual set 210 of text samples 212, each text sample 212 including a monolingual text representation of a respective sentence. At operation 404, the method 400 generates, for each text sample 212 in the multilingual set 210 of text samples 212, a modified synthetic version 222 of the respective text sample 212. The modified synthetic version 222 of the respective text sample 212 includes grammatical changes to the monolingual text representation of the respective sentence associated with the respective text sample 212. At operation 406, the method 400 trains the text generation model 122 using the training set 230 of training sample pairs 232. Each pair 232 in the training set 230 of sample pairs 232 includes one of the respective text samples 212 in the multilingual set 210 of text samples 212 and a corresponding modified synthetic version 222 of that one of the respective text samples 212 in the multilingual set 210 of text samples 212.

[0028] 5 is a schematic diagram of an exemplary computing device 500 that may be used to implement the systems (e.g., NLG system 120 and / or GEC model 124) and methods (e.g., method 400) described herein. Computing device 500 is intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other suitable computers. The components shown here, their connections and relationships, and their functionality are intended to be exemplary only and are not intended to limit the implementations described and / or claimed herein.

[0029] Computing device 500 includes processor 510 (e.g., data processing hardware 112, 144), memory 520 (e.g., memory hardware 114, 146), storage device 530, high-speed interface / controller 540 connecting to memory 520 and high-speed expansion port 550, and low-speed interface / controller 560 connecting to low-speed bus 570 and storage device 530. Each of components 510, 520, 530, 540, 550, and 560 are interconnected using various buses and may be attached to a common motherboard or otherwise as needed. Processor 510 can process instructions for execution within computing device 500, including instructions stored in memory 520 or storage device 530, to display graphical information for a graphical user interface (GUI) on an external input / output device, such as display 580 coupled to high-speed interface 540. In other implementations, multiple processors and / or multiple buses may be used, along with multiple memories and types of memory, as needed. Also, multiple computing devices 500 may be connected, each providing a portion of the required operations (eg, as a bank of servers, a group of blade servers, or a multi-processor system).

[0030] The memory 520 stores information non-temporarily within the computing device 500. The memory 520 may be a computer-readable medium, a volatile memory unit, or a non-volatile memory unit. The non-volatile memory 520 may be a physical device used to temporarily or permanently store programs (e.g., a sequence of instructions) or data (e.g., program state information) for use by the computing device 500. Examples of non-volatile memory include, but are not limited to, flash memory and read-only memory (ROM) / programmable read-only memory (PROM) / erasable programmable read-only memory (EPROM) / electrically erasable programmable read-only memory (EEPROM) (e.g., commonly used for firmware such as boot programs). Examples of volatile memory include, but are not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), phase change memory (PCM), and disk or tape.

[0031] The storage device 530 is capable of providing mass storage for the computing device 500. In some implementations, the storage device 530 is a computer-readable medium. In various different implementations, the storage device 530 may be a floppy disk device, a hard disk device, an optical disk device, or an array of devices including a tape device, a flash memory or other similar solid-state memory device, or a storage area network or other configuration of devices. In further implementations, a computer program product is tangibly embodied on an information carrier. The computer program product includes instructions that, when executed, perform one or more methods, such as those described above. The information carrier is a computer-readable or machine-readable medium, such as the memory 520, the storage device 530, or memory on the processor 510.

[0032] The high-speed controller 540 manages the bandwidth-intensive operations of the computing device 500, and the low-speed controller 560 manages the lower bandwidth-intensive operations. This allocation of duties is merely exemplary. In some implementations, the high-speed controller 540 is coupled to the memory 520, the display 580 (e.g., via a graphics processor or accelerator), and is also coupled to a high-speed expansion port 550 that may accept various expansion cards (not shown). In some implementations, the low-speed controller 560 is coupled to the storage device 530 and a low-speed expansion port 570. The low-speed expansion port 570 may include various communication ports (e.g., USB, Bluetooth, Ethernet, wireless Ethernet) and may be coupled to one or more input / output devices such as a keyboard, pointing device, scanner, etc., or to a networking device such as a switch or router, for example, via a network adapter.

[0033] The computing device 500 may be implemented in several different forms, as shown in the figure. For example, the computing device 500 may be implemented as a standard server 500a, or multiple times in a group of such servers 500a, or as a laptop computer 500b, or as part of a rack server system 500c.

[0034] Various implementations of the systems and techniques described herein may be realized in digital electronic and / or optical circuitry, integrated circuits, specially designed ASICs (application-specific integrated circuits), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include implementation in one or more computer programs executable and / or interpretable by a programmable system including at least one programmable processor, which may be special purpose or general purpose, coupled to receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.

[0035] These computer programs (also known as programs, software, software applications, or code) include machine instructions for a programmable processor and may be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, non-transitory computer-readable medium, apparatus, and / or device (e.g., magnetic disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0036] The processes and logic flows described herein may be implemented by one or more programmable processors executing one or more computer programs to perform functions by operating on input data and generating output. The processes and logic flows may also be implemented by special purpose logic circuitry, such as an FPGA (field programmable gate array) or an ASIC (application-specific integrated circuit). Processors suitable for executing computer programs include, by way of example, both general-purpose and special purpose microprocessors, and any one or more processors of any type of digital computer. A processor will generally receive instructions and data from a read-only memory or a random-access memory, or both. The essential elements of a computer are a processor for executing instructions and one or more memory devices for storing instructions and data. A computer will also generally include one or more mass storage devices, such as magnetic, magneto-optical, or optical disks, for storing data, or be operably coupled to receive data from or transfer data to them, or both. However, a computer need not have such devices. Computer-readable media suitable for storing computer program instructions and data include all forms of non-volatile memory, media, and memory devices, including, by way of example, semiconductor memory devices such as EPROM, EEPROM, and flash memory devices, magnetic disks such as internal hard disks or removable disks, magneto-optical disks, and CD-ROM and DVD-ROM disks. The processor and memory may be supplemented by, or incorporated in, special purpose logic circuitry.

[0037] To enable interaction with a user, one or more aspects of the present disclosure may be implemented in a computer having a display device, such as a CRT (cathode ray tube), LCD (liquid crystal display) monitor, or touch screen, for displaying information to the user, and optionally a keyboard and pointing device, such as a mouse or trackball, by which the user can provide input to the computer. Other types of devices may also be used to enable interaction with the user; for example, feedback provided to the user may be any form of sensory feedback, such as visual, auditory, or tactile feedback, and input from the user may be received in any form, including acoustic, voice, or tactile input. Additionally, the computer may interact with the user by sending documents to and receiving documents from devices used by the user, for example, by sending web pages to a web browser on the user's client device in response to a request received from the web browser.

[0038] Although several implementations have been described, it will be understood that various modifications may be made without departing from the spirit and scope of the present disclosure. Accordingly, other implementations are within the scope of the following claims. [Explanation of symbols]

[0039] 10 users 100 Text Correction Environment 110 User Devices 112 Data Processing Hardware 114 Memory Hardware 116 Audio Subsystem 118 Applications 120 Natural Language Generation (NLG) Systems 122 Text Generation Model 124 Grammatical Error Correction (GEC) Model 126 Text Sequences 128 Output Text 130 Network 140 Remote Systems 142 Remote Resources 144 Remote Data Processing Hardware 146 Remote Memory Hardware 200 Training Process 210 Corpus 212 Text Samples 220 Corrupta 222 Modified Synthetic Version / Text Version / Text Sample 230 training set / training dataset 232 training sample pairs 300 Fine-tuning Process 310 Fine-tuned Training Corpus 312 Non-composite text pairs 314 Grammatically Correct Text Samples 316 grammatically incorrect text samples 500 computing devices 510 processor 520 memory 530 Storage Devices 540 High-Speed ​​Interface / Controller 550 High-Speed ​​Expansion Port 560 Low-Speed ​​Interface / Controller 570 low-speed bus / low-speed expansion port 580 Display

Claims

1. A computer-implemented method (400) for training a text generation model (122) for grammatical error correction (GEC), the method, when executed by data processing hardware (144), causing the data processing hardware (144) to: obtaining a multilingual set (210) of text samples (212), each text sample (212) comprising a monolingual text representation of a respective sentence; generating, for each text sample (212) of the multilingual set (210) of text samples (212), a modified synthetic version (222) of the respective text sample (212), the modified synthetic version (222) of the respective text sample (212) including grammatical modifications to the monolingual text representation of the respective sentence associated with the respective text sample (212), the grammatical modifications being non-language-specific to a particular language of a plurality of languages ​​included in the multilingual set to prevent bias against any particular types of grammatical errors that are specific to the particular language; training the text generation model (122) using a training set (230) of sample pairs (232), wherein each sample pair in the training set (230) of sample pairs (232) one of said respective text samples (212) of said multilingual set (210) of text samples (212); a corresponding modified synthetic version (222) of one of said respective text samples (212) of said multilingual set (210) of text samples (212); a training step, A method (400) for performing an operation including:

2. 10. The method of claim 1, wherein the operations further comprise, after training the text generation model, fine-tuning the trained text generation model using supervised training data, the supervised training data including non-synthetic text pairs, each non-synthetic text pair including a grammatically incorrect text sample and a grammatically correct text version of the ungrammatical text sample.

3. 3. The method of claim 1, wherein generating the modified synthesized version of each text sample comprises removing two or more characters from the respective sentence associated with the respective text sample.

4. 2. The method of claim 1, wherein generating the modified synthesized version of the respective text sample comprises replacing a first set of characters from the respective sentence associated with the respective text sample with a second set of characters that differ from the first set of characters.

5. 2. The method of claim 1, wherein generating the modified synthesized version of the respective text sample comprises inserting one or more characters into the respective sentence associated with the respective text sample.

6. 2. The method of claim 1, wherein generating the modified synthesized version of each text sample comprises changing the letter case of letters in words of the respective sentence associated with the respective text sample.

7. Generating the modified synthetic version (222) of each of the text samples (212) comprises randomly applying a modification operation to each of the sentences associated with each of the text samples (212), the modification operation comprising: removing two or more characters from each of the sentences associated with each of the text samples (212); replacing a first set of characters from each sentence associated with each text sample (212) with a second set of characters different from the first set of characters; inserting one or more characters into the respective sentence associated with the respective text sample (212); or changing the case of letters of words in each of said sentences associated with each of said text samples (212); and 10. The method of claim 1, wherein each modified synthesized version is unique with respect to other modified synthesized versions of the text sample.

8. 10. The method of claim 1, wherein the text generation model comprises a transformer encoder-decoder architecture.

9. 10. The method of claim 1, wherein the operations further comprise pre-training the text generation model using a masked language objective-based multilingual training corpus.

10. After training the text generation model (122) for GEC, the trained text generation model (122) receiving as input a first input text (126) in a first language containing grammatical errors and producing as output a first output text (128) in said first language that corrects said grammatical errors; receiving as input a second input text (126) in a different second language containing grammatical errors, and generating as output from the trained text generation model (122) a second output text (128) in the second language that corrects the grammatical errors; The method (400) of claim 1, configured to:

11. data processing hardware (144); memory hardware (146) in communication with the data processing hardware (144), the memory hardware (146) executing on the data processing hardware (144) causing the data processing hardware (144) to: obtaining a multilingual set (210) of text samples (212), each text sample (212) comprising a monolingual text representation of a respective sentence; generating, for each text sample (212) of the multilingual set (210) of text samples (212), a modified synthetic version (222) of the respective text sample (212), the modified synthetic version (222) of the respective text sample (212) including grammatical modifications to the monolingual text representation of the respective sentence associated with the respective text sample (212), the grammatical modifications being non-language-specific to a particular language of a plurality of languages ​​included in the multilingual set to prevent bias against any particular types of grammatical errors that are specific to the particular language; training a text generation model (122) using a training set (230) of sample pairs (232), wherein each sample pair (232) in the training set (230) of sample pairs (232) one of said respective text samples (212) of said multilingual set (210) of text samples (212); a corresponding modified synthetic version (222) of one of said respective text samples (212) of said multilingual set (210) of text samples (212); a training step, the memory hardware (146) storing instructions for performing operations including A system (100) comprising:

12. 12. The system of claim 11, wherein the operations further comprise, after training the text generation model, fine-tuning the trained text generation model using supervised training data, the supervised training data including non-synthetic text pairs, each non-synthetic text pair including a grammatically incorrect text sample and a grammatically correct text version of the ungrammatical text sample.

13. 13. The system (100) of claim 11 or 12, wherein generating the modified synthesized version (222) of the respective text sample (212) comprises removing two or more characters from the respective sentence associated with the respective text sample (212).

14. 12. The system of claim 11, wherein generating the modified synthesized version of the respective text sample includes replacing a first set of characters from the respective sentence associated with the respective text sample with a second set of characters that are different from the first set of characters.

15. 12. The system of claim 11, wherein generating the modified synthesized version of the respective text sample comprises inserting one or more characters into the respective sentence associated with the respective text sample.

16. 12. The system of claim 11, wherein generating the modified synthesized version of the respective text sample comprises changing the case of letters in words of the respective sentences associated with the respective text sample.

17. Generating the modified synthetic version (222) of each of the text samples (212) comprises randomly applying a modification operation to each of the sentences associated with each of the text samples (212), the modification operation comprising: removing two or more characters from each of the sentences associated with each of the text samples (212); replacing a first set of characters from each sentence associated with each text sample (212) with a second set of characters different from the first set of characters; inserting one or more characters into the respective sentence associated with the respective text sample (212); or changing the case of letters of words in each of said sentences associated with each of said text samples (212); and 12. The system (100) of claim 11, wherein each modified synthesized version (222) is unique with respect to other modified synthesized versions (222) of the text sample (212).

18. 12. The system of claim 11, wherein the text generation model comprises a transformer encoder-decoder architecture.

19. 12. The system (100) of claim 11, wherein the operations further comprise pre-training the text generation model (122) using a masked language objective-based multilingual training corpus (210).

20. After training the text generation model (122) for GEC, the trained text generation model (122) receiving as input a first input text (126) in a first language containing grammatical errors and producing as output a first output text (128) in said first language that corrects said grammatical errors; receiving as input a second input text (126) in a different second language containing grammatical errors, and generating as output from the trained text generation model (122) a second output text (124) in the second language that corrects the grammatical errors; The system (100) of claim 11 configured to:

Images