A method, system, readable medium, and device for masked machine translation

Abstract

The application provides a masked machine translation method, system, readable medium and equipment, comprising: sorting training instance data according to source sequences from long to short, establishing a new order, and distributing target sequences according to corresponding source sequence indexes; taking the maximum length x of the source sequences and the maximum length y of the target sequences as the maximum limit lengths of two independent storage spaces respectively; putting source sequence pairs into the first storage space in the current order one by one, when a row in the first storage space is not filled, determining the remaining size of the current row, traversing the current order, if there is source sequence data less than or equal to the remaining size, splicing the source sequence data and the data that has been filled, updating the order, and storing the target sequence data corresponding to the source sequence data into the second storage space; and until any storage space is full, forming new training instance data. The application improves the efficiency while maintaining the original performance of the model.

Description

Technical Field

[0001] This invention belongs to the field of machine translation technology, specifically relating to a masked machine translation method, system, readable medium, and device. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] The input to current machine translation tasks, whether sentences, paragraphs, or documents, varies in length. Current machine translation training heavily relies on high-performance deep learning infrastructure, such as the PyTorch deep learning framework and the A100 hardware processor, both of which process batches of data of uniform length. Since truncating input data leads to information loss and impairs translation performance, the standard practice is to pad the encoded natural language data to the longest possible sequence length before feeding it into the model for training. Because the padding characters are typically set to zero, this padding strategy is commonly known as zero-padding. Due to the complex and diverse nature of natural languages, the padded batches contain a large number of padding characters. Currently, mainstream datasets such as WMT14 (English to German) and WMT17 (Chinese to English) have padding rates exceeding 60% during training. The model needs to perform masking operations to avoid the influence of these padding characters on the actual language characters. These padding characters have no practical meaning during training, and the large number of padding characters increases training time and wastes computational resources.

[0004] Given the inefficiency of machine translation tasks, common methods include bucketing and sorting strategies. Sorting strategies rank data by length and then feed it into the model for training. This significantly reduces the fill rate, but it also alters the data distribution, meaning the model uses length-biased data instead of random data during training. Therefore, while this method significantly shortens training time, it violates the principle of unbiased estimation in model training, thus harming model performance. Bucketing generates several buckets based on the data length distribution, with each bucket containing sequences of different lengths. Data is then divided into the appropriate buckets based on length, and training is performed on the data within each bucket separately. This approach is a compromise; while it reduces the fill rate to some extent, it also impacts model performance. Summary of the Invention

[0005] To address the aforementioned problems, this invention proposes a masked machine translation method, system, readable medium, and device. This invention overcomes the drawback of inefficient training caused by high fill rate in machine translation tasks, and maintains the original performance of the model while improving efficiency.

[0006] According to some embodiments, the present invention adopts the following technical solution:

[0007] A masked machine translation method includes the following steps:

[0008] The sentence is translated based on the trained translation model;

[0009] The training process of the translation model includes:

[0010] The training instance data is sorted from longest to shortest according to the source sequence to establish a new order, and the target sequence is distributed according to the corresponding source sequence index.

[0011] The maximum length x of the source sequence and the maximum length y of the target sequence are respectively used as the maximum limit lengths of two independent storage spaces;

[0012] The source sequence pairs are placed into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined. The current order is traversed. If there is a source sequence data that is less than or equal to the remaining size, it is concatenated with the data that has already been filled and the sorting order is updated. The target sequence data corresponding to the source sequence data is stored in the second storage space.

[0013] Until any storage space is full, new training instance data is generated.

[0014] As an alternative implementation, the maximum length x of the source sequence and the maximum length y of the target sequence may not correspond to the same sentence pair.

[0015] As an alternative implementation, if there is no training instance data in the current order that is less than or equal to the remaining size length, the corresponding row of the storage space is padded to the longest sequence length.

[0016] As an alternative implementation, if there is source sequence data with a length less than or equal to the remaining size, and the length of the concatenated target sequence data after concatenating it with the already filled data exceeds the maximum length y of the target sequence, then the current data concatenation is not performed, and the corresponding row of the storage space is filled to the longest sequence length.

[0017] As a further implementation, zeros are added after the current position when filling.

[0018] As an alternative implementation, the following steps are also included:

[0019] During model training, when generating the mask matrix, mask labels are marked for different sequences, and attention regions that intersect between different sequences are masked before softmax processing.

[0020] As a further implementation, the same masking operation is performed only on the generated diagonal matrix.

[0021] A masked machine translation system, comprising:

[0022] The translation module is configured to translate sentences based on a trained translation model;

[0023] The training module includes:

[0024] The sorting module is configured to sort the training instance data from longest to shortest according to the source sequence, establish a new order, and distribute the target sequence according to the corresponding source sequence index.

[0025] The limiting module is configured to use the maximum length x of the source sequence and the maximum length y of the target sequence as the maximum limiting lengths of two independent storage spaces, respectively.

[0026] The splicing module is configured to put source sequence pairs into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined, the current order is traversed, and if there is source sequence data with a length less than or equal to the remaining size, it is spliced ​​with the data that has already been filled, the sorting order is updated, and the target sequence data corresponding to the source sequence data is stored in the second storage space.

[0027] The iterative module is configured to generate new training instance data until any storage space is full.

[0028] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.

[0029] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.

[0030] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0031] This invention uses the input of the next training instance data to fill a short input, forming a mixed instance input, and uses the lengths of the source sequence and the target sequence as a constraint on storage space to pack as many elements as possible in each batch, which can effectively improve training speed and reduce the filling rate.

[0032] This invention is based on the bin packing problem, taking into account the attributes of both the source and target sequences. It finds the longest source and target sequences respectively, and uses this pair of data as the size of the bin. It flexibly considers the case where the longest source and target sequences do not belong to the same pair, and constructs a way to maximize sequence filling.

[0033] This invention allows users to define the size of training instance data and storage space, and to splice data in multiple batches without interference between batches. This ensures that the model gradient accumulation is the same as the standard training method, and also ensures that the sorting within the batch does not affect the calculation of the model's cross-entropy loss, without requiring additional parameter adjustments to the model.

[0034] This invention addresses the issue of cross-interference during attention calculation in spliced ​​hybrid instances by employing a novel attention masking strategy. Without altering the internal structure of the transformer, it ensures that the attention between each sequence does not interfere with each other, achieving efficient training of masked hybrid instances and obtaining significant acceleration without sacrificing training quality.

[0035] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings. Attached Figure Description

[0036] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0037] Figure 1 This is a diagram illustrating the attention mechanism;

[0038] Figure 2 This is a schematic diagram of the packing process of the present invention. Detailed Implementation

[0039] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0041] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0042] Example 1

[0043] As described in the background section, in current machine translation tasks, almost all NLP tasks require a padding strategy (zero-padding) to align different training instances (within a batch) to the same size, and it is common for training data to have a large proportion of padding characters. Therefore, zero-padding strategies are very wasteful, sometimes resulting in most of the training input (after padding) being filled with meaningless padding marks, ultimately harming training efficiency.

[0044] This embodiment proposes a novel machine translation training padding method, called the instance fusion strategy. The basic idea is not to use zero-padding, but to use the input of the next training instance data to pad a short input, forming a "fusion instance input," where multiple training instance data are concatenated together.

[0045] A masked machine translation method includes the following steps:

[0046] The sentence is translated based on the trained translation model;

[0047] The training process of the translation model includes:

[0048] The training instance data is sorted from longest to shortest according to the source sequence to establish a new order, and the target sequence is distributed according to the corresponding source sequence index.

[0049] The maximum length x of the source sequence and the maximum length y of the target sequence are respectively used as the maximum limit lengths of two independent storage spaces;

[0050] The source sequence pairs are placed into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined. The current order is traversed. If there is a source sequence data that is less than or equal to the remaining size, it is concatenated with the data that has already been filled and the sorting order is updated. The target sequence data corresponding to the source sequence data is stored in the second storage space.

[0051] If there are no training instance data in the current order that are less than or equal to the remaining size length, then fill the corresponding row of the storage space to the length of the longest sequence.

[0052] If there is a source sequence data with a length less than or equal to the remaining size, but the length of the corresponding target sequence data after concatenation with the already filled data exceeds the maximum length y of the target sequence, then the current data concatenation will not be performed, and the corresponding row in the storage space will be filled up to the longest sequence length.

[0053] Continue the above splicing process until any storage space is full, forming new training instance data.

[0054] The process described above essentially involves rearranging instances within the same training batch to create a more compact mix of instances, ensuring that the entire batch is filled with as many real-world language data as possible. This rearrangement problem can be formulated as a 2D bin packing problem: each training instance is an element with its own two-dimensional size (the lengths of the source and target sequences); given a set of such elements (instances in the entire training dataset), the goal is to pack as many elements as possible into each batch.

[0055] Specifically, for ease of understanding by technical personnel, we'll use bin packing as an analogy. We'll assume the storage space is a rectangular bin, and each word is a rectangular item. The two-dimensional bin packing problem involves packing several rectangular items into rectangular bins (bins), and during the packing process, the rectangular items cannot be placed diagonally. Generally, the goal is to minimize the number of bins used. In this problem, each item contains two attributes, each with a size. The given bins also contain two attributes. The goal is to fill all items with the fewest bins possible. The two-dimensional bin packing problem is like filling a large rectangle with several small rectangles.

[0056] The constraint here is that the sum of the lengths and widths of all items is less than or equal to the length and width of the bin. The maximum sequence length in this embodiment (i.e., the bin in the bin packing problem) can be modified, and the optimization objective is to find a way to maximize the filling of the entire batch, which differs from the standard bin packing problem.

[0057] In this embodiment, both source and target sequence attributes are considered. At the beginning of the algorithm, the longest source and target sequences are found, and this pair of data is used as the size of the bin. It is worth noting that, due to the characteristics of natural language, the longest sequence size sometimes does not belong to the same translation pair. This method flexibly considers the case where the longest source and target sequences do not belong to the same sentence pair, constructing a way to maximize sequence padding.

[0058] like Figure 2 As shown, for the convenience of those skilled in the art, the following example is provided:

[0059] Sort the source sequences by length, starting with the longest and proceeding to the shortest. The length of the longest statement in the source sequence is x, and the length of the longest statement in the target sequence is y.

[0060] Construct two boxes with lengths x and y respectively.

[0061] The first bin is used to hold the source sequence, with the longest statement in the first line. The second longest statement is placed in the second line onwards. If there are more statements in the source sequence after the second longest statement is placed, then... Figure 2For the 8 empty slots shown, we traverse the current sort to find a sentence of length 8. Then, we use a greedy algorithm to traverse from top to bottom (from longest to shortest) until we find a sentence that matches the length and whose length after translation is less than y. Then, we extract the sentence and concatenate the two sentences.

[0062] Next, the third row is loaded. If there is an empty space after loading, and there is a sequence that meets the length of the empty space, but the concatenation length of the two target sequences is greater than y, then the concatenation of the two sequences is abandoned, and zero padding is performed on the row.

[0063] Alternatively, if there is no sequence shorter than the empty space length, abandon the splicing and pad the row with zeros.

[0064] A single line can contain the concatenation of multiple (more than two) sequences.

[0065] During model training, the attention computation of concatenated mixed instances introduces cross-interference. A novel attention masking strategy is proposed to address this issue. When generating the mask matrix, we label different sequences with mask tags. Consistent with the standard transformer, we mask the intersecting attention regions between different sequences before softmax processing.

[0066] like Figure 1 As shown, A, B, and C represent the attention mechanisms of the standard transformer within the encoder and decoder, respectively, while C, D, and E represent the masked attention mechanism of the hybrid instance designed in this invention. The hybrid instance uses the concatenation of two sequences as an example, leaving a diagonal matrix after masking. Each blue matrix represents the self-attention of its respective sentence, indicating the relevance of each clause. Similarly, in the masked multi-head attention layer of the decoder module, the standard transformer masks the target sequence into an upper triangle.

[0067] This embodiment simply performs the same masking operation on the generated diagonal matrix. The proposed method, without altering the internal structure of the transformer, ensures that the attention between each sequence does not interfere with each other, achieving efficient training of masked mixed instances.

[0068] By comparison, the method provided in this embodiment can reduce the fill rate from 69% to 13% in WMT14 English to German translation and from 66% to 5% in WMT17 Chinese to English translation. The reduction in fill rate translates to a training time reduction of more than 2 times.

[0069] Table 1. Model performance and time on WMT14 and WMT17 datasets

[0070]

[0071]

[0072] As shown in Table 1, the experimental results of the baseline model Transformer (TRM) and two existing popular methods for bucketing and sorting are presented, along with the masked hybrid instance method (source sequence aligned hybrid (src), target sequence aligned hybrid (tgt), and source-target 2D hybrid (src-tgt)) proposed in this invention. The BLEU score serves as the benchmark for machine translation tasks, with higher scores indicating better performance. In the WMT14 English-to-German translation task, the method in this embodiment significantly reduces both the fill rate and training time. The model performance obtained in this embodiment is essentially consistent with the standard Transformer. In the WMT17 Chinese-to-English translation task, the method in this embodiment is more efficient in reducing the fill rate, simultaneously reducing the fill rate of both the source and target sequences to below 7% in the 2D hybrid instance method. This demonstrates wider applicability.

[0073] The training time of this embodiment is similar to that of the sorting method, and the training effect is better than both methods. This shows that the method significantly improves the efficiency of model training while ensuring performance. Finally, all four experiments maintained model accuracy, indicating that the masked instance mixing method avoids attention interference between sequences.

[0074] The present invention also provides the following product examples:

[0075] A masked machine translation system, comprising:

[0076] The translation module is configured to translate sentences based on a trained translation model;

[0077] The training module includes:

[0078] The sorting module is configured to sort the training instance data from longest to shortest according to the source sequence, establish a new order, and distribute the target sequence according to the corresponding source sequence index.

[0079] The limiting module is configured to use the maximum length x of the source sequence and the maximum length y of the target sequence as the maximum limiting lengths of two independent storage spaces, respectively.

[0080] The splicing module is configured to put source sequence pairs into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined, the current order is traversed, and if there is source sequence data with a length less than or equal to the remaining size, it is spliced ​​with the data that has already been filled, the sorting order is updated, and the target sequence data corresponding to the source sequence data is stored in the second storage space.

[0081] The iterative module is configured to generate new training instance data until any storage space is full.

[0082] A computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the steps in the above method.

[0083] An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, wherein the computer instructions, when executed by the processor, perform the steps in the method described above.

[0084] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0086] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0087] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1The steps of the function specified in one or more boxes.

[0088] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made by those skilled in the art without creative effort within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A masked machine translation method, characterized in that, Includes the following steps: The sentence is translated based on the trained translation model; The training process of the translation model includes: The training instance data is sorted from longest to shortest according to the source sequence to establish a new order, and the target sequence is distributed according to the corresponding source sequence index. The maximum length x of the source sequence and the maximum length y of the target sequence are respectively used as the maximum limit lengths of two independent storage spaces; The source sequence pairs are placed into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined. The current order is traversed. If there is a source sequence data that is less than or equal to the remaining size, it is concatenated with the data that has already been filled and the sorting order is updated. The target sequence data corresponding to the source sequence data is stored in the second storage space. Until any storage space is full, new training instance data is generated.

2. The masked machine translation method as described in claim 1, characterized in that, The maximum length x of the source sequence and the maximum length y of the target sequence may not correspond to the same sentence pair.

3. The masked machine translation method as described in claim 1, characterized in that, If there are no training instance data in the current order that are less than or equal to the remaining size length, then fill the corresponding row of the storage space to the length of the longest sequence.

4. The masked machine translation method as described in claim 1, characterized in that, If there is a source sequence data with a length less than or equal to the remaining size, and the length of the concatenated target sequence data after concatenating it with the already filled data exceeds the maximum length y of the target sequence, then the current data concatenation will not be performed, and the corresponding row in the storage space will be filled up to the longest sequence length.

5. A masked machine translation method as described in claim 3 or 4, characterized in that, When filling, add zeros after the current position.

6. The masked machine translation method as described in claim 1, characterized in that, It also includes the following steps: During model training, when generating the mask matrix, mask labels are marked for different sequences, and attention regions that intersect between different sequences are masked before softmax processing.

7. The masked machine translation method as described in claim 6, characterized in that, The same masking operation is performed only on the generated diagonal matrix.

8. A masked machine translation system, characterized in that, include: The translation module is configured to translate sentences based on a trained translation model; The training module includes: The sorting module is configured to sort the training instance data from longest to shortest according to the source sequence, establish a new order, and distribute the target sequence according to the corresponding source sequence index. The limiting module is configured to use the maximum length x of the source sequence and the maximum length y of the target sequence as the maximum limiting lengths of two independent storage spaces, respectively. The splicing module is configured to put source sequence pairs into the first storage space in the current order. When a row in the first storage space is not full, the remaining size of the current row is determined, the current order is traversed, and if there is source sequence data with a length less than or equal to the remaining size, it is spliced ​​with the data that has already been filled, the sorting order is updated, and the target sequence data corresponding to the source sequence data is stored in the second storage space. The iterative module is configured to generate new training instance data until any storage space is full.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, complete the steps of the method according to any one of claims 1-7.

10. An electronic device, characterized in that, It includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the steps of the method according to any one of claims 1-7.

Images