Length-constrained machine translation models

Abstract

Aspects of the present disclosure are directed to controlling machine translation length based on length tokens that are included in the machine translated source text and the target text during training and in the machine translated source text during inference, and where output from a length-constrained machine learning model is generated if the output from an unconstrained machine learning model outputs a translation that exceeds the length limit.

Description

[Background technology]

[0001] Machine translation corresponds to the use of software to translate text or speech from one language to another. In some scenarios, such as advertising, user interfaces, or voice-overs, it may be desirable to control the length of the machine translation output. Length normalization, verbosity tokens, and positional coding have been used to control the length of the machine translation output. However, with length normalization, it may be difficult to match the number of characters, display width, and / or speech duration. Furthermore, length normalization may have a small effect on the length output when implementing a greedy beam search. With verbose tokens, the categorical nature of the tokens (e.g., short, normal, long) makes it difficult to precisely control the translation length. With positional coding, only the number of tokens can be controlled, but it may be desirable to control the number of characters or display width. Character-level vocabulary can be implemented, but this significantly increases latency. Furthermore, model output using positional coding produces translations with strict length constraints, as opposed to outputs below the length constraint. Summary of the Invention [Means for solving the problem]

[0002] Aspects of the present disclosure are directed to controlling machine translation length based on length tokens. Length tokens are included in machine translation source texts and target texts during training so that machine learning models can learn the length of each token. Length tokens are also included in machine translation source texts during inference so that machine learning models can output length-constrained translations. If an output from an unconstrained machine learning model outputs a translation that exceeds the length limit, a subsequent output is generated from a length-constrained machine learning model. If the subsequent output still exceeds the length limit, another output is generated from a length-constrained machine learning model with a reduced length limit. Machine translation length control can be implemented in advertisements, headlines and / or descriptions in user interface messages on mobile or other computing devices, or dubbing translations for movies or television. Aspects of the present disclosure may therefore provide improved machine translation, particularly in implementations where limitations are imposed on the fields or contexts in which translated text is stored, output, or displayed. By controlling the machine translation length as described herein, information that might otherwise be lost if, for example, the translated text were to exceed a length limit can instead be retained, stored, communicated, or displayed to a user.

[0003] An aspect of the present disclosure provides a method for length-constrained machine translation, the method including receiving, by one or more processors, data corresponding to a source text, translating, by the one or more processors, the source text using an unconstrained length machine learning model to generate data corresponding to a first translated text, determining, by the one or more processors, that the first translated text exceeds a length limit, translating, by the one or more processors, the source text using a length-constrained machine learning model to generate data corresponding to a second translated text, and outputting, by the one or more processors, the data corresponding to the second translated text.

[0004] In one example, the method further includes determining, by the one or more processors, that the second translated text exceeds the text length limit; and decreasing, by the one or more processors, the length limit for the length-constrained machine learning model, wherein the length limit for the length-constrained machine learning model is iteratively decreased until the translated text translated using the length-constrained machine learning model does not exceed the length limit.

[0005] In another example, the method further includes adding, by the one or more processors, a length token to the beginning of the source text to represent the length constraint. In yet another example, the method further includes estimating, by the one or more processors, a length of the first translated text. In yet another example, the method further includes increasing, with the one or more processors, a randomness value of the length constraint.

[0006] In yet another example, the method further includes training, using one or more processors, the length-constrained machine learning model using training data including a plurality of pairs of source text and translated text, each pair having one or more length tokens added to it. In yet another example, the source text of each pair includes a length token added to the beginning of the source text to represent the text length limit. In yet another example, the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the text length limit. In yet another example, the method further includes merging, using one or more processors, the training data with training data for an unconstrained machine learning model.

[0007] Another aspect of the present disclosure provides a system including one or more processors and one or more storage devices coupled to the one or more processors that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for length-constrained machine translation, including receiving data corresponding to a source text, translating the source text using an unconstrained length-machine learning model to generate data corresponding to a first translated text, determining that the first translated text exceeds a length limit, translating the source text using a length-constrained machine learning model to generate data corresponding to a second translated text, and outputting the data corresponding to the second translated text.

[0008] In one example, the operations further include determining that the second translated text exceeds a text length limit and decreasing the length limit for the length-constrained machine learning model, wherein the length limit for the length-constrained machine learning model is iteratively decreased until the translated text translated using the length-constrained machine learning model does not exceed the length limit.

[0009] In another example, the operations further include adding a length token to the beginning of the source text to represent the length constraint. In yet another example, the operations further include estimating a length of the first translated text. In yet another example, the operations further include increasing a randomness value of the length constraint.

[0010] In yet another example, the operations further include training the length-constrained machine learning model using training data including a plurality of pairs of source text and translated text, each pair having one or more length tokens added to it. In yet another example, the source text of each pair includes a length token added to the beginning of the source text to represent the text length limit. In yet another example, the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the text length limit. In yet another example, the operations further include merging the training data with training data for an unlength-constrained machine learning model.

[0011] Yet another aspect of the present disclosure provides a non-transitory computer-readable medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for length-constrained machine translation, including receiving data corresponding to a source text, translating the source text using an unconstrained length machine learning model to generate data corresponding to a first translated text, determining that the first translated text exceeds a length limit, translating the source text using a length-constrained machine learning model to generate data corresponding to a second translated text, and outputting the data corresponding to the second translated text.

[0012] In one example, the operations further include determining that the second translated text exceeds a text length limit and decreasing the length limit for the length-constrained machine learning model, wherein the length limit for the length-constrained machine learning model is iteratively decreased until the translated text translated using the length-constrained machine learning model does not exceed the length limit.

[0013] In another example, the operations further include adding a length token to the beginning of the source text to represent the length constraint. In yet another example, the operations further include estimating a length of the first translated text. In yet another example, the operations further include increasing a randomness value of the length constraint.

[0014] In yet another example, the operations further include training the length-constrained machine learning model using training data including a plurality of pairs of source text and translated text, each pair having one or more length tokens added to it. In yet another example, the source text of each pair includes a length token added to the beginning of the source text to represent the text length limit. In yet another example, the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the text length limit. In yet another example, the operations further include merging the training data with training data for an unlength-constrained machine learning model. [Brief explanation of the drawings]

[0015] [Figure 1] FIG. 1 is a block diagram of an exemplary length-constrained machine translation system according to aspects of the present disclosure. [Figure 2] FIG. 1 is a block diagram of an example environment for implementing a length-constrained machine translation system according to aspects of the present disclosure. [Figure 3] FIG. 1 is a block diagram of an exemplary machine translation model architecture according to aspects of the present disclosure. [Figure 4]1 is a flow diagram of an exemplary process for training a machine translation model according to an aspect of the present disclosure. [Figure 5] 1 is a flow diagram of an exemplary process for length-constrained machine translation according to an aspect of the present disclosure. [Figure 6] FIG. 1 is a block diagram of an example length-constrained machine translation system in which unconstrained and constrained machine learning models may be integrated, according to aspects of the present disclosure. DETAILED DESCRIPTION OF THE INVENTION

[0016] Generally, disclosed herein are implementations for controlling machine translation length based on length tokens. Length tokens are added to machine translation source texts and target texts during training so that a machine learning model can learn the length of each token. The source text may correspond to the text to be translated. In the source text, a length token is inserted at the beginning of the text to indicate the desired length constraint. The target text may correspond to a translation of the source text. In the target text, one or more length tokens are inserted in the translated text to indicate the remaining length constraint. Machine translation length control can be implemented in advertisements, headlines and / or descriptions in user interface messages on mobile or other computing devices, or dubbing of translations.

[0017] To train a machine learning model, the training data may include source-translation pairs as well as other metadata such as timestamps, component types, productions, etc. A sentence fragment model (SPM) may include a mapping from tokens to identifiers and may be trained using the training data. Based on output from the SPM model, the source-translation pairs in the training data may be tokenized and converted into identifiers to generate tokenized training data. The machine learning model may be trained using the tokenized training data.

[0018] To control the length of the machine translation, length tokens may be inserted into the source text and the target text so that the machine learning model is aware of the length information. To train the machine learning model to take length into account, the training data may further include length constraints.

[0019] The length constraint may correspond to an actual length constraint or a pseudo length constraint. The actual length constraint may correspond to a predetermined length limit included with the training data. The pseudo length constraint may correspond to a length limit that is the length of the translated text when the training data does not include the actual length constraint. The length constraint may be expressed numerically and may, for example, limit the number of characters, words, or phrases.

[0020] Length tokens may be added to the source text and the target text to indicate length constraints. The length tokens may be added to the beginning of the source text. One or more length tokens may be added to the target text after each tokenized text element, such as after each word token, to indicate the remainder of the length constraint. For example, the remainder may be represented numerically to indicate the number of remaining characters, words, or phrases that can be generated.

[0021] The length constraint can further include a randomness value during model training, because forcing the machine learning model to output the exact same length as opposed to a length between ranges would not maintain translation quality. The addition of the randomness value gives the machine learning model some flexibility in generating translation lengths. The length flexibility can be represented by a hyperparameter.

[0022] For model inference, the model can tokenize the source text and then convert it into a source identifier. The source identifier may be input to a trained machine learning model, which can output a target identifier after encoding and decoding. The target identifier may be converted into a machine translation and detokenized.

[0023] To control the length of the machine translation, model inference is similar to training. For example, a length token may be added to the beginning of the source text. Model inference differs in that it uses a hybrid approach, described below, to generate length-constrained machine translation output and repeatedly retry model inference with decreasing length constraints.

[0024] Given a length constraint, the machine learning model should produce a translation whose length is equal to or less than the length constraint. A hybrid approach is used, where a translation is generated by a length-unconstrained machine learning model, and if the translation exceeds the length limit, the translation is generated again by a length-constrained machine learning model. Furthermore, to increase accuracy, if the translation still exceeds the length limit in the length-constrained machine learning model, the length limit is reduced and the length-constrained machine learning model is run again. The reduction of the length limit is repeated until the translation conforms to the length.

[0025] The process for length-constrained machine translation inference may include several variations to integrate the unconstrained model and the constrained model to reduce complexity. For example, a target length for machine translation may be estimated before running model inference. The target length may be estimated by rules or by a model. As another example, the length flexibility may be increased to a value greater than the length limit. To alleviate the sparseness problem, the training data may include some randomly generated examples in addition to actual examples. As yet another example, the distribution of length flexibility may be varied. As yet another example, the training data of the unconstrained model and the length-constrained model may be merged. Here, the machine learning model may learn that if the first token is not a length token, an unconstrained translation may be output, but if the first token is a length token, a constrained translation may be output.

[0026] 1 shows a block diagram of an exemplary length-constrained machine translation system 100. The length-constrained machine translation system 100 may be configured to receive input data, including training data 102 and inference data 104, via a user interface. For example, the length-constrained machine translation system 100 may receive the input data as part of a call to an API that exposes the length-constrained machine translation system 100. The length-constrained machine translation system 100 may be implemented on one or more computing devices. Input to the length-constrained machine translation system 100 may be provided through a storage medium, including remote storage, connected to one or more computing devices via a network, or as input through a user interface on a client computing device coupled to the length-constrained machine translation system 100.

[0027] The length-constrained machine translation system 100 may be configured to receive training data 102 for training a machine learning model in translation and inference data 104 that specifies a target translation. The training data 102 may correspond to a machine learning task related to translation, such as a neural network task performed by a neural network. The training data 102 may be divided into a training set, a validation set, and / or a test set. An exemplary training / test split may be an 80 / 20 split. The machine learning model may be configured to receive any type of input data to generate output data 106 for performing a machine learning task related to translation. By way of example, the output data 106 may be any type of score, classification, or regression output that translates the input data. Accordingly, the machine learning task may be a scoring, classification, and / or regression task related to translation. These machine learning tasks may correspond to a variety of different applications in processing image, video, text, audio, or other types of data for translation.

[0028] The training data 102 may be in any form suitable for training a machine learning model according to one of a variety of different learning techniques. Learning techniques for training a machine learning model may include supervised learning, unsupervised learning, and semi-supervised learning techniques. For example, the training data 102 may include a plurality of training examples that may be received as input by the machine learning model. The training examples may be labeled with a desired output for the machine learning model when processing the labeled training examples. The labels and model output may be evaluated through a loss function to determine an error, and the error may be back-propagated through the machine learning model to update the weights for the machine learning model. For example, if the machine learning task is a classification task, the training examples may be images labeled with one or more classes that categorize the subject matter depicted in the image. As another example, a supervised learning technique may be applied to calculate the error between the output and the ground truth labels of the training examples processed by the machine learning model. Any of a variety of loss or error functions may be used that are appropriate for the type of task for which the machine learning model is being trained, such as cross-entropy loss for classification or mean squared error for regression. A gradient of the error for different weights of the candidate model on the candidate hardware can be calculated, for example, using a backpropagation algorithm, and the weights for the model can be updated. The machine learning model can be trained until a stopping criterion is met, such as a number of iterations for training, a maximum time period, convergence, or when a minimum accuracy threshold is met.

[0029] The training data 102 may include source-translation pairs as well as metadata such as timestamps, component types, productions, etc. The source-translation pairs can be tokenized and converted into identifiers to generate tokenized training data based on the mappings. The mappings may be determined, for example, by a sentence fragment model (SPM). A machine learning model can be trained for translation using the tokenized training data.

[0030] For example, the source text of a source-translation pair may be "cheap rental cars Miami." The source text may be tokenized into ["_cheap", "_rental", "_cars", "_Mi", "ami", ""] and then converted into a list of identifiers [8174,6509,6984,602,5943,2]. "_" can represent a word boundary, and " <s> " can indicate the beginning of a sentence in the source text, and "< / s> " may indicate the end of a sentence in the target text of a source-translation pair. Low frequency words such as "Miami" may be split into subwords.

[0031] The training data 102 may include length constraints to train the machine learning model to take length into account when translating. For example, a source-translation pair may be formatted as (source, length_constraint) → translation. The length constraint may correspond to an actual length constraint, such as a predetermined length limit included with the training data 102, or a pseudo-length constraint, such as the length of the target text in the source-translation pair. The length constraint may be expressed numerically and may, for example, limit the number of characters, words, or phrases. For example, "Nice to meet you!" →

number

number

number

number

[0032] The source-translation pairs in the training data 102 may include length tokens in the source and target texts to represent length constraints. For example, a length token may be represented as "TOKxx," where "xx" is the length constraint. The source text may include a length token before the text to be translated. The target text may include one or more length tokens after each tokenized text element in the translated text to indicate the remainder of the length constraint. By way of example, the remainder may be represented numerically to indicate the number of remaining characters, words, or phrases that can be generated.

[0033] Below are example source-translation pairs of training data 102 for a machine learning model for translating from English to Spanish when the pseudo-length constraint is 49: Source_word:[" <tok49>", "_He", "_looks", "_around", ",", "_seemingly", "_un", "sure", "_of", "_where", "_he", "_is", ".", ""] Target_word:[" <s>", "_Mira", " <tok45>", "_around", " <tok35>", "_how", " <tok30>", "_yes", "TOK27>", "_no", " <tok24>", "_sup", " <tok20>", "iera", " <tok16>", "_where", " <tok10>", "_se", " <tok7>", "_tilt", " <tok1> ", ".", " <tok0>"]

[0034] The pseudo-length constraint is 49, so the length token " <tok49>" is inserted in the source text. In the target text, <tokxx>represents the number of remaining characters that can be generated. So after outputting "_Mira", the remaining characters are reduced to 45, and after outputting "_alrededor", the remaining characters are reduced again to 35. This is repeated until all tokens have been output. The pseudo-length constraint is equal to the length of the target text, so the final token in target_word is " <tok0>" should be, indicating that no characters remain.

[0035] The length constraints in the training data 102 can further include a randomness value, as forcing the machine learning model to output the exact same length, as opposed to a length between ranges, would not maintain translation quality. The addition of the randomness value gives the machine learning model some flexibility in generating translation lengths.

[0036] For example, the length constraint can be expressed as length_constraint=len(target)+uniform_random(0,R), where R is a hyperparameter that represents the length flexibility of the machine learning model. For example, if the length constraint is 50 characters and R is 10, the machine learning model can output translation lengths between 40 and 50 characters. Increasing R allows the machine learning model to generate a wider range of translation lengths, but also increases the sparseness of the training data 102, which may increase the difficulty for the machine learning model to learn the length constraint.

[0037] The inference data 104 may correspond to data to be translated based on a machine learning model trained on the training data 102. The inference data 104 may include source text as well as other metadata such as a timestamp, component type, production, etc. The source text of the inference data 104 may include a length token at the beginning of the text to be translated. For example, the length token may be represented as "TOKxx," where "xx" represents a length constraint.

[0038] The source text may be tokenized and converted into identifiers to generate tokenized inference data based on the mapping. The mapping may be determined by an SPM, for example. The tokenized inference data may be input to a trained machine learning model to output a target identifier. The target identifier may be converted into a machine translation and de-tokenized. The output data 106 may correspond to the machine translation. The output data may also correspond to the target identifier to be converted into a machine translation by another computing device. Length-constrained machine translation can be performed using a hybrid approach including unconstrained machine learning and length-constrained machine learning models, as well as repeatedly retrying model inference with decreasing length constraints.

[0039] From the training data 102 and the inference data 104, the length-constrained machine translation system 100 may be configured to output one or more results of a machine learning task related to translation, generated as output data 106. The output data 106 may be sent for display on a user display, for example. In some implementations, the length-constrained machine translation system 100 may be configured to provide the output data 106 as a set of computer-readable instructions, such as one or more computer programs. The computer programs may be written in any type of programming language and may follow any programming paradigm, for example, declarative, procedural, assembly, object-oriented, data-oriented, functional, or imperative. The computer programs may be written to perform one or more different functions and to operate within a computing environment, for example, on a physical device, a virtual machine, or across multiple devices. The computer programs may also implement functionality described herein, for example, performed by a system, engine, module, or model.

[0040] The length-constrained machine translation system 100 may be configured to forward the output data 106 to one or more other devices configured to convert the output data 106 into an executable program written in a computer programming language. The length-constrained machine translation system 100 may also be configured to send the output data 106 to a storage device for storage and subsequent retrieval.

[0041] The length-constrained machine translation system 100 may include an unconstrained length engine 108. The unconstrained length engine 108 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination of the foregoing. The unconstrained length engine 108 may be configured to generate machine translations from the training data 102 and / or the inference data 104 using a machine learning model that is not length-constrained.

[0042] The length-constrained machine translation system 100 may further include a length constraint engine 110. The length constraint engine 110 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination of the foregoing. The length constraint engine 110 may be configured to determine whether the machine translation produced by the unconstrained length engine 108 exceeds a length constraint. The length constraint engine 110 may compare the length of the machine translation output from the unconstrained length machine learning model with a predetermined length constraint. If the machine translation is equal to or less than the predetermined length limit, the machine translation may be output as the output data 106. If the machine translation is greater than the predetermined length limit, the machine translation is not output.

[0043] The length-constrained machine translation system 100 may also include a constraint length engine 112. The constraint length engine 112 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination of the foregoing. The constraint length engine 112 may be configured to generate a machine translation from the training data 102 and / or the inference data 104 using a machine learning model constrained by a length limit. If the machine translation generated by the unconstrained length engine 108 exceeds the length limit, the machine translation is generated again by the constraint length engine 112.

[0044] The length constraint engine 110 may also be configured to determine whether the machine translation generated by the constraint length engine 112 exceeds a length constraint. The length constraint engine 110 may compare the length of the machine translation output from the length-constrained machine learning model to a predetermined length constraint. If the machine translation is equal to or less than the predetermined length constraint, the machine translation may be output as output data 106.

[0045] If the machine translation is larger than the predetermined length limit, the machine translation is not output. Instead, the length limit engine 110 may decrease the length limit for the length-constrained machine learning model, for example, by one character, word, or phrase. The constraint length engine 112 may generate a subsequent machine translation from the training data 102 and / or the inference data 104 using the machine learning model constrained by the decreased length limit. The length limit engine 110 may determine whether the subsequent machine translation exceeds the original length limit. The length limit engine 110 may iteratively decrease the length limit for the length-constrained machine learning model, and the constraint length engine 112 may iteratively generate machine translations based on the iteratively decreasing length limit until the generated machine translation complies with the length limit, such as by being equal to or smaller than the original length limit.

[0046] 2 shows a block diagram of an exemplary environment 200 for implementing a length-constrained machine translation system. The system 200 can be implemented on one or more devices having one or more processors in one or more locations, such as in a server computing device 202. The client computing device 204 and the server computing device 202 may be communicatively coupled to one or more storage devices 206 via a network 208. The storage device 206 may be a combination of volatile and non-volatile memory and may be in the same or a different physical location as the computing devices 202, 204. For example, the storage device 206 may include any type of non-transitory computer-readable medium capable of storing information, such as a hard drive, a solid-state drive, a tape drive, optical storage, a memory card, ROM, RAM, DVD, CD-ROM, writable, and read-only memory.

[0047] The server computing device 202 may include one or more processors 210 and memory 212. The memory 212 may store information accessible by the processor 210, including instructions 214 that may be executed by the processor 210. The memory 212 may also include data 216 that may be retrieved, manipulated, or stored by the processor 210. The memory 212 may be a type of non-transitory computer-readable medium capable of storing information accessible by the processor 210, such as volatile and non-volatile memory. The processor 210 may include one or more central processing units (CPUs), graphics processing units (GPUs), field programmable gate arrays (FPGAs), and / or application specific integrated circuits (ASICs), such as tensor processing units (TPUs).

[0048] The instructions 214 may include one or more instructions that, when executed by the processor(s) 210, cause one or more processors to perform actions defined by the instructions 214. The instructions 214 may be stored in object code format for direct processing by the processor(s) 210, or in other formats, including an interpretable script or a collection of independent source code modules that are interpreted on-demand or pre-compiled. The instructions 214 may include instructions for implementing a length-constrained machine translation system 218, which may correspond to the length-constrained machine translation system 100 of FIG. 1. The length-constrained machine translation system 218 may be executed using the processor 210 and / or using other processors located remotely from the server computing device 202.

[0049] Data 216 may be retrieved, stored, or modified by processor 210 in accordance with instructions 214. Data 216 may be stored in computer registers, in a relational or non-relational database, as a table with multiple distinct fields and records, or as a JSON, YAML, proto, or XML document. Data 216 may be formatted in a computer-readable format, such as, but not limited to, binary values, ASCII, or Unicode. Moreover, data 216 may include relevant information such as numbers, descriptive text, unique codes, pointers, references to data stored in other memory, including other network locations, or information sufficient to identify information used to calculate the relevant data by a function.

[0050] The client computing device 204 may be configured similarly to the server computing device 202, having one or more processors 220, memory 222, instructions 224, and data 226. The client computing device 204 may also include a user input 228 and a user output 230. The user input 228 may include any suitable mechanism or technique for receiving input from a user, such as a keyboard, a mouse, a mechanical actuator, a soft actuator, a touchscreen, a microphone, and a sensor.

[0051] The server computing device 202 may be configured to transmit data to the client computing device 204, which may be configured to display at least a portion of the received data on a display implemented as part of the user output 230. The user output 230 may also be used to display an interface between the client computing device 204 and the server computing device 202. The user output 230 may alternatively or additionally include one or more speakers, transducers or other audio outputs, haptic interfaces or other tactile feedback that provide non-visual and non-auditory information to a platform user of the client computing device 204.

[0052] While FIG. 2 depicts the processors 210, 220 and memories 212, 222 as being within the computing devices 202, 204, the components described herein may include multiple processors and memories that may operate in different physical locations and not within the same computing device. For example, some of the instructions 214, 224 and data 216, 226 may be stored on a removable SD card, while other instructions and data may be stored in a read-only computer chip. Some or all of the instructions and data may be stored in locations physically separate from the processors 210, 220 but still accessible by those processors. Similarly, the processors 210, 220 may include a collection of processors that may perform simultaneous and / or sequential operations. The computing devices 202, 204 may each include one or more internal clocks that provide timing information that may be used to time operations and programs run by the computing devices 202, 204.

[0053] Server computing device 202 may be connected via network 208 to a data center 232 that houses any number of hardware accelerators 232A-N. Data center 232 may be one of multiple data centers or other facilities that house various types of computing devices, such as hardware accelerators. Computational resources housed in data center 232 may be designated for deploying machine learning models for translation, as described herein.

[0054] The server computing device 202 may be configured to receive requests to process data from the client computing devices 204 on computational resources in the data center 232. For example, the environment 200 may be part of a computing platform configured to provide various services to users through various user interfaces and / or application programming interfaces (APIs) that expose platform services. The various services may include generating one or more machine learning models for translation. The client computing device 204 may transmit data specifying text to be translated along a length constraint for the translated text. The length-constrained machine translation system 218 may receive the data specifying the text to be translated and the length constraint and, in response, generate output data including translated text that conforms to the length constraint.

[0055] As another example of possible services provided by a platform implementing environment 200, server computing device 202 may maintain various machine learning models according to different possible length restrictions or translation details available at data center 232. For example, server computing device 202 may maintain different families for deploying neural networks on various types of TPUs and / or GPUs housed at data center 232 or otherwise available for processing.

[0056] 3 shows a block diagram 300 illustrating one or more machine translation model architectures 302, more specifically, each architecture 302A-N, for deployment in a data center 304 that houses a hardware accelerator 306 on which the deployed machine translation model 302 will execute to provide translations constrained by length limits. The hardware accelerator 306 may be any type of processor, such as a CPU, GPU, FPGA, or ASIC such as a TPU.

[0057] The machine translation model architecture 302 can refer to characteristics that define the model, such as characteristics of layers for the model, how the layers process input, or how the layers interact with each other. The machine translation model architecture 302 may be for a length-unconstrained machine learning model or a length-constrained machine learning model. The machine translation model architecture 302 may correspond to an encoder-decoder architecture, such as a transformer. Input data, such as input text, may be parsed into tokens, for example, by a byte-pair encoding tokenizer. Each token may be converted into a vector, such as by word embedding.

[0058] An encoder may include a coding layer that iteratively processes input data layer by layer, while a decoder includes a decoding layer that iteratively processes the encoder's output data layer by layer. Each encoder layer can generate an encoding that includes information about which portions of the input data are related to each other. Each encoder layer then sends its encoding as input to the next encoder layer. Each decoder layer can consider all of the encodings and use their context information to generate an output sequence. For each portion of the input, an attention unit, such as a scaled dot-product attention unit, can weigh the relevance of each other portion and generate an output from them. Each decoder layer may have an additional attention mechanism to derive information from the output of the previous decoder before the decoder layer derives information from the encoding. Both the encoder and decoder layers may include a feedforward neural network for additional processing of the output, as well as residual connections and layer normalization steps.

[0059] One or more machine translation model architectures 302 can be generated that are capable of outputting translation results that comply with length constraints.

[0060] Referring again to FIG. 2 , the devices 202, 204 and the data center 232 may be capable of direct communication and indirect communication via the network 208. For example, using a network socket, the client computing device 204 can connect to a service running in the data center 232 through the Internet Protocol. The devices 202, 204 can set up a listening socket that can accept an initiating connection to send and receive information. The network 208 itself may include a variety of configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local network, and a private network using one or more company-specific communication protocols. The network 208 can support a variety of short- and long-range connections. The short- and long-range connections may occur over different bandwidths, such as 2.402 GHz to 2.480 GHz, commonly associated with the Bluetooth® standard, or 2.4 GHz and 5 GHz, commonly associated with the Wi-Fi® communication protocol, or with various communication standards, such as the LTE® standard for wireless broadband communication. The network 208 may additionally or alternatively support wired connections between the devices 202, 204 and the data center 232, including via various types of Ethernet connections.

[0061] 2, it will be understood that aspects of the present disclosure may be implemented according to a variety of different configurations and quantities of computing devices, including in paradigms for sequential or parallel processing, or via a distributed network of multiple devices. In some implementations, aspects of the present disclosure may be performed on a single device connected to a hardware accelerator configured to process machine learning models, and any combination thereof.

[0062] 4 shows a flow diagram of an example process 400 for training a machine translation model. The example process 400 may be implemented on a system of one or more processors in one or more locations, such as the length-constrained machine translation system 100 of FIG.

[0063] As shown in block 410, the unconstrained length engine 108 and / or the constrained length engine 112 may receive training data 102. The training data 102 may include source-translation pairs. The training data 102 for the unconstrained length engine 108 may not include a length constraint, while the training data 102 for the constrained length engine 112 may include a length constraint. The length constraint may correspond to an actual length constraint or a pseudo-length constraint. The length constraint may further include a randomness value to add flexibility to the machine translation length.

[0064] As shown in block 420, the unconstrained length engine 108 and / or the constrained length engine 112 can tokenize and convert the training data 102. The unconstrained length engine 108 can convert the training data 102 into identifiers based on a mapping, which can be determined by the SPM. The constrained length engine 112 can tokenize the training data 102 by inserting length tokens into the source text and target text of a source-translation pair to represent a length constraint. The source text may include a length token before the text to be translated. The target text may include one or more length tokens after each tokenized text element of the translated text to indicate the remainder of the length constraint. The constrained length engine 112 can then convert the tokenized training data 102 into identifiers based on a mapping, which can be determined by the SPM.

[0065] As shown in block 430, the unconstrained length engine 108 can train an unconstrained machine learning model for translation based on the training data 102 converted into identifiers. The unconstrained length engine 108 can train the unconstrained machine learning model using various learning techniques, including supervised learning, unsupervised learning, or semi-supervised learning. The unconstrained machine learning model may correspond to a Transformer having an encoder-decoder architecture.

[0066] As shown in block 440, the constraint length engine 112 can train a length-constrained machine learning model for translation based on the tokenized and identifier-converted training data 102. The constraint length engine 112 can train the length-constrained machine learning model using various learning techniques, including supervised learning, unsupervised learning, or semi-supervised learning. The constrained machine learning model may correspond to a Transformer having an encoder-decoder architecture.

[0067] 5 shows a flow diagram of an exemplary process 500 for length-constrained machine translation. The exemplary process 500 may be implemented on a system of one or more processors in one or more locations, such as the length-constrained machine translation system 100 of FIG.

[0068] As shown in block 510, the unconstrained length engine 108 may receive inference data 104. The inference data 104 may include source text with a length token at the beginning of the text to be translated to represent a length constraint.

[0069] As indicated in block 520, the unconstrained length engine 108 may perform machine translation using an unconstrained machine learning model to generate the translation.

[0070] As shown in block 530, the length constraint engine 110 may determine whether the translation from the unconstrained machine learning model exceeds the length constraint. The length constraint engine 110 may compare the length of the translation from the unconstrained machine learning model to the length constraint. If the translation is less than or equal to the length constraint, the translation may be output.

[0071] If the translation is greater than the length constraint, the constrained length engine 112 may perform machine translation using a constrained machine learning model to generate another translation, as shown in block 540. The constrained machine learning model may output additional length tokens in its output translation compared to the unconstrained machine learning model output. If additional length tokens are output, they may be removed in post-processing.

[0072] As shown in block 550, the length constraint engine 110 may determine whether the translation from the constrained machine learning model exceeds the length constraint. The length constraint engine 110 may compare the length of the translation from the constrained machine learning model to the length constraint. If the translation is less than or equal to the length constraint, the translation may be output.

[0073] If the translation is larger than the length limit, the length limit engine 110 may decrease the length limit, as shown in block 560. For example, the length limit engine 110 may decrease the length limit by one character, word, or phrase.

[0074] As shown in block 570, the constraint length engine 112 can perform a subsequent machine translation using the constrained machine learning model with the reduced length constraint to generate another translation. The length constraint engine 110 can determine whether the subsequent translation from the constrained machine learning model exceeds the original length constraint. If the translation is equal to or less than the original length constraint, the translation may be output. If the translation is greater than the original length constraint, the length constraint engine 110 can again reduce the length constraint, for example, by one character, word, or phrase. The length constraint engine 110 and the constraint length engine 112 can iteratively reduce the length constraint and perform machine translation using the constrained machine learning model until the translation complies with the original length constraint.

[0075] The overall process for length-constrained machine translation inference can be further described as follows. Inputs: source text source, length constraint L, unconstrained MT model m1, length-constrained MT model m2, maximum number of attempts max_retry_count. Output: Length-constrained MT output. mt=runInference(m1,source) current_limit=L while len(mt)>L and max_retry_count>=0: mt=runInference(m2,source,current_limit) current_limit=current_limit-1 max_retry_count=max_retry_count-1 return mt

[0076] 6 shows a block diagram of an exemplary length-constrained machine translation system 600 in which unconstrained and constrained machine learning models may be integrated to reduce complexity. Training data 602, inference data 604, and output data 606 may correspond to training data 102, inference data 104, and output data 106 of FIG. 1.

[0077] The length-constrained machine translation system 600 may include a combined constrained / unconstrained length engine 608. The combined length engine 608 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination of the foregoing. The combined length engine 608 may be configured to train a machine learning model for length-constrained translation and to perform length-constrained translation using the trained machine learning model.

[0078] The length-constrained machine translation system 600 may further include a length constraint engine 610. The length constraint engine 610 may be implemented as one or more computer programs, specially configured electronic circuitry, or any combination of the foregoing.

[0079] For example, the length constraint engine 610 may be configured to estimate a target length for machine translation before running model inference in the composite length engine 608. The length constraint engine 610 may estimate the target length by rules or by models. For rule-based estimation, the length constraint engine 610 may estimate the target length by, for example, multiplying the length of the source text by a predetermined factor. For model-based estimation, the length constraint engine 610 estimates the target length by running a machine learning model for length estimation.

[0080] As another example, the length constraint engine 610 can increase the length flexibility to a value greater than the length constraint. To mitigate the sparseness problem, the training data 602 can include some randomly generated examples in addition to the actual examples.

[0081] As yet another example, the composite length engine 608 can vary the distribution of length flexibility, where training data 602 can be generated using lower and higher length flexibility values, and the composite length engine 608 can merge two training data sets.

[0082] As yet another example, the composite length engine 608 can merge the training data 602 of the unconstrained model and the length-constrained model, where the machine learning model can learn that if the first token is not a length token, an unconstrained translation can be output as the output data 606, but if the first token is a length token, a constrained translation can be output as the output data 606.

[0083] Aspects of the present disclosure may be implemented as digital circuitry, one or more computer programs in a computer-readable storage medium, or a combination of one or more of the above. The computer-readable storage medium may be non-transitory, for example, as one or more instructions executable by a cloud computing platform and stored on a tangible storage device.

[0084] The phrase "configured to" is used in different contexts in reference to a computer system, hardware, or part of a computer program. When a system is said to be configured to perform one or more operations, this means that the system has appropriate software, firmware, and / or hardware installed thereon, which, when operating, causes the system to perform one or more operations. When some hardware is said to be configured to perform one or more operations, this means that the hardware includes one or more circuits, which, when operating, receive inputs and generate outputs according to the inputs and corresponding to the one or more operations. When a computer program is said to be configured to perform one or more operations, this means that the computer program includes one or more program instructions, which, when executed by one or more computers, cause the one or more computers to perform one or more operations.

[0085] Unless otherwise specified, the above alternatives are not mutually exclusive but may be implemented in various combinations to achieve particular advantages. As these and other variations and combinations of the features discussed above can be used without departing from the subject matter defined by the claims, the above description of embodiments should be taken as examples and not as limitations on the subject matter defined by the claims. Furthermore, the provision of examples described herein, as well as clauses expressed as "such as," "including," and the like, should not be construed as limiting the subject matter of the claims to specific examples, which are intended to illustrate only one of many possible embodiments. Furthermore, the same reference numbers in different drawings may identify the same or similar elements. [Explanation of symbols]

[0086] 100 Length-Constrained Machine Translation System 108 Unconstrained Length Engine 110 Length-Limited Engine 112 Constraint Length Engine 200 Environment, System 202 Server computing device, computing device, device 204 Client Computing Device, Computing Device, Device 206 Storage Devices 208 Network 210 processor 212 memory 218 Length-Constrained Machine Translation System 220 processors 222 memory 228 User Input 230 User Output 232 Data Center 232A~N Hardware Accelerator 302 Machine Translation Model, Machine Translation Model Architecture 304 Data Center 306 Hardware Accelerator 600 Length-Constrained Machine Translation System 608 Composite Constrained / Unconstrained Length Engine, Composite Length Engine 610 Length Limited Engine < / tokxx> < / tok1> < / s>

Claims

1. 1. A method for length-constrained machine translation, comprising: receiving, by one or more processors, data corresponding to the source text; translating, by the one or more processors, the source text using an unconstrained length machine learning model to generate data corresponding to a first translated text; determining, by the one or more processors, that the first translated text exceeds a length limit; translating, by the one or more processors, the source text using a length-constrained machine learning model to generate data corresponding to a second translated text; outputting, by the one or more processors, the data corresponding to the second translated text; the method further comprising using the one or more processors to increase the randomness value of the length limit; method.

2. determining, by the one or more processors, that the second translated text exceeds the length limit; and reducing, by the one or more processors, a length limit for the length-constrained machine learning model; 2. The method of claim 1, wherein the length limit for the length-constrained machine learning model is iteratively decreased until translated text translated using the length-constrained machine learning model does not exceed the length limit.

3. The method of claim 1 or claim 2, further comprising adding, by the one or more processors, a length token to the beginning of the source text to represent the length restriction.

4. The method of claim 1 or 2, further comprising the step of estimating, by the one or more processors, a length of the first translated text.

5. A method as described in claim 1 or 2, further comprising a step of training the length-constrained machine learning model using the one or more processors with training data including multiple pairs of source text and translated text, each with one or more length tokens added.

6. The method of claim 5 , wherein the source text of each pair includes a length token added to the beginning of the source text to indicate the length restriction.

7. The method of claim 5 , wherein the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the length constraint.

8. 6. The method of claim 5, further comprising using the one or more processors to merge the training data with training data for the machine learning model that is not length constrained.

9. one or more processors; and one or more storage devices coupled to the one or more processors that store instructions that, when executed by the one or more processors, cause the one or more processors to perform operations for length-constrained machine translation, the operations comprising: receiving data corresponding to a source text; translating the source text using an unconstrained length machine learning model to generate data corresponding to a first translated text; determining that the first translated text exceeds a length limit; translating the source text using a length-constrained machine learning model to generate data corresponding to a second translated text; outputting the data corresponding to the second translated text; The operations further include increasing the randomness value of the length limit. system.

10. The operation is determining that the second translated text exceeds the length limit; and reducing a length limit for the length-constrained machine learning model; 10. The system of claim 9, wherein the length limit for the length-constrained machine learning model is iteratively decreased until translated text translated using the length-constrained machine learning model does not exceed the length limit.

11. 11. The system of claim 9 or 10, wherein the actions further include adding a length token to the beginning of the source text to represent the length restriction.

12. The system of claim 9 or 10, wherein the operations further include estimating a length of the first translated text.

13. The system described in claim 9 or 10, wherein the operation further includes training the length-constrained machine learning model using training data including multiple pairs of source text and translated text, each with one or more length tokens added.

14. 14. The system of claim 13, wherein the source text of each pair includes a length token added to the beginning of the source text to indicate the length restriction.

15. 14. The system of claim 13, wherein the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the length constraint.

16. 14. The system of claim 13, wherein the operations further include merging the training data with training data for the machine learning model that is not length constrained.

17. 1. A computer-readable storage medium for storing instructions that, when executed by one or more processors, cause the one or more processors to perform operations for length-constrained machine translation, the operations comprising: receiving data corresponding to a source text; translating the source text using an unconstrained length machine learning model to generate data corresponding to a first translated text; determining that the first translated text exceeds a length limit; translating the source text using a length-constrained machine learning model to generate data corresponding to a second translated text; outputting the data corresponding to the second translated text; the operations further include increasing the randomness value of the length limit. A computer-readable storage medium.

18. The operation is determining that the second translated text exceeds the length limit; and reducing a length limit for the length-constrained machine learning model; 18. The computer-readable storage medium of claim 17, wherein the length limit for the length-constrained machine learning model is iteratively decreased until translated text translated using the length-constrained machine learning model does not exceed the length limit.

19. 19. The computer-readable storage medium of claim 17 or 18, wherein the operations further comprise adding a length token to the beginning of the source text to represent the length restriction.

20. 19. The computer-readable storage medium of claim 17 or 18, wherein the operations further include estimating a length of the first translated text.

21. A computer-readable storage medium as described in claim 17 or 18, wherein the operation further includes training the length-constrained machine learning model using training data including multiple pairs of source text and translated text, each with one or more length tokens added.

22. 22. The computer-readable storage medium of claim 21, wherein the source text of each pair includes a length token added to the beginning of the source text to represent the length restriction.

23. 22. The computer-readable storage medium of claim 21, wherein the translated text of each pair includes one or more length tokens added after each tokenized text element to represent the remainder of the length constraint.

24. 22. The computer-readable storage medium of claim 21, wherein the operations further comprise merging the training data with training data for the machine learning model that is not length constrained.

Images