A method and system for forming dynamically sized training batches for a neural machine translation system

Abstract

The present invention discloses a method and system for forming a dynamic size training batch of a neural machine translation system, which is based on the field of neural machine translation. The method comprises: parsing parallel corpora, segmenting and vectorizing the corpora to obtain vectorized sentence pairs; dividing the vectorized sentence pairs into blocks, and sorting the sentence pairs in the vectorized sentence pair blocks in ascending order according to the number of terms; grouping and filling the sorted sentence pairs in the sentence pair blocks according to the batch size, wherein the batch size is the sum of the lengths of all sentence pairs in a batch, so as to form a training batch. The present invention dynamically adjusts the training batch size according to the number of sentence terms in the training of the neural machine translation system, and after sorting, the short sentence pairs in the sentence pair block are placed in front, and the sentence pairs with similar lengths are adjacent. Since the sentence pairs with similar number of terms are adjacent in position in the block after sorting, the sentence pairs in each group after grouping are relatively close in the number of terms, and the sentence pairs that need to be filled and the length of the filling are less. Under the same video memory, a training batch with a larger average length can be formed, thereby improving the training efficiency and effect.

Description

Technical Field

[0001] The present invention belongs to the field of neural machine translation based on deep learning, and more specifically, to a method and system for forming a dynamic-sized training batch for a neural machine translation system. Background Art

[0002] Neural machine translation has now become the de facto standard for machine translation and has achieved the best machine translation results so far. The translation results of neural machine translation systems are highly dependent on training on large-scale parallel corpora. Dividing parallel corpora into multiple mini-batches and performing back propagation and gradient descent are currently common techniques used in the training of various neural machine translation systems.

[0003] At present, the basic training method of neural machine translation system is to first calculate the loss of the system on the training batch by forward reasoning on the mini batch or training batch, then reversely calculate the gradient of the loss to the model parameters, and finally update the model parameters according to the gradient. Usually, a parallel corpus is divided into multiple training batches, and the neural machine translation system is iteratively trained on the training batch of the corpus. This is an approximate training method, which approximates the expected loss on the training data with the loss on the training batch. In this process, the size of the training batch has a great impact on the training effect and training efficiency. The larger the training batch, the more the model can make full use of the parallelism of the GPU, the faster the training speed, and the more accurate the gradient estimation of the loss calculated on the training batch to the parameters, the faster the model converges, and the easier it is for the training to find the approximate point of the optimal result. Therefore, as long as the hardware allows, various neural machine translation systems will use the largest possible training batch. In addition, in order to ensure the training effect, the training corpus needs to be shuffled before being divided into training batches, so the sentence lengths in the same training batch are often different, while the general neural machine translation system requires the data length in the training batch to be consistent, which is usually achieved by padding the sentences in the training batch to a uniform length. Therefore, training batches with the same number of sentences and consistent sentence length padding are the basic approach for training batches in current neural machine translation systems.

[0004] Although the training batches with the same number of sentences and consistent sentence length padding make the preparation of training data for the neural machine translation system simpler and easier to implement, this training batch formation method has the following problems: First, the original lengths of the shuffled sentences are different, and the lengths may vary greatly. Using padding to achieve consistent length means that the neural machine translation system's processing of a lot of data in each batch is meaningless (because there is a lot of meaningless content padding), which reduces the actual effective throughput of GPU training and also reduces the training speed; secondly, the length of the training batch with the same number of sentences and consistent sentence length padding is determined by the length of the longest sentence. Even if the lengths of many sentences are much smaller than the longest sentence, they must be padded consistently, which affects the size of the final training batch, that is, the longer the longest sentence, the smaller the training batch will be or the smaller the number of sentences it contains. A too small training batch will make the gradient estimate of the updated weights inaccurate, and the neural machine translation model converges more slowly, so that either a longer training time is required to achieve acceptable performance, or it is more difficult to approach the global optimum. Summary of the invention

[0005] In view of the defects and improvement needs of poor machine translation model training efficiency and training effect caused by training batches with the same number of sentences in existing neural machine translation systems, the present invention provides a method and system for forming dynamic size training batches of a neural machine translation system, which aims to dynamically adjust the training batch size according to the number of sentence entries. The training batch can include more sentence pairs on average, that is, the average batch size is larger, which can reduce the training convergence time of the neural machine translation system and improve the training efficiency.

[0006] To achieve the above objective, according to a first aspect of the present invention, a method for forming a dynamic size training batch for a neural machine translation system is provided, the method comprising:

[0007] S1. Parse the parallel corpus, segment and vectorize the corpus, and obtain vectorized sentence pairs;

[0008] S2. Divide the vectorized sentence pairs into blocks, and sort the sentence pairs in the vectorized sentence pair blocks in ascending order according to the number of terms;

[0009] S3. Grouping and padding the sorted sentence pairs in the sentence pair block according to the batch size, where the batch size is the sum of the lengths of all sentence pairs in a batch, to form a training batch.

[0010] Preferably, step S1 comprises:

[0011] S11. Parse the parallel corpus to obtain source language and target language sentence convection;

[0012] S12. Segment the sentences in the sentence stream, generate source language and target language dictionaries and save them;

[0013] S13. Use the generated source language and target language dictionaries to vectorize the parallel corpus.

[0014] Beneficial effects: In view of the diversity of formats and language pairs of existing corpora, the present invention realizes flexible support for vectorized processing of multiple formats and multiple corpora by dynamically configuring dictionaries and maintaining the mapping from corpus formats to parsers and from languages ​​to segmenters.

[0015] Preferably, in step S11, parallel corpora in different formats use different parsers, and the correspondence between the corpus formats and the parsers is stored in a configuration file.

[0016] Beneficial effects: In view of the problem that there are currently multiple different parallel corpus formats, the present invention maps the parallel corpus format to the corpus parser through dynamic configuration, thereby flexibly supporting multiple parallel corpus parsing.

[0017] Preferably, in step S12, different text languages ​​use different segmenters, and the correspondence between the text languages ​​and the segmenters is stored in a configuration file.

[0018] Beneficial effects: In order to solve the segmentation problem of hundreds of language pairs, the present invention configures a dictionary to maintain the mapping between languages ​​and segmenters, and uses a built-in default segmenter to flexibly support the segmentation of parallel corpora of multiple languages.

[0019] Preferably, in step S13, if the size of the generated dictionary is specified, a counting structure of entries and their number of occurrences in the text is maintained, the number of occurrences is updated during the scanning of the text, and the counting structure is sorted in descending order according to the number of occurrences after the scanning is completed, and the entries of the previously specified dictionary size in the sorted counting structure are used as dictionary content.

[0020] Beneficial effects: Fixed dictionary sizes are difficult to adapt to different corpus sizes and different workstation computing capabilities. The present invention counts and sorts the occurrence frequencies of segmented terms, and selects more frequently appearing terms as dictionary items according to the specified dictionary size, thereby allowing convenient and flexible selection of appropriate dictionary pairs for vectorization of sentence pairs.

[0021] Preferably, in step S2, the vectorized sentence pairs are cached in blocks, one vectorized sentence pair block is taken each time, and the sentence pairs in the vectorized sentence pair block are sorted in ascending order according to the total length of the two digital sequences.

[0022] Beneficial effects: As the parallel corpus becomes larger and larger, batching all the vectorized sentence pairs in memory will consume more and more memory, and may even become impossible. The present invention processes the vectorized sentence pairs in blocks and batches according to the machine resource configuration. The size of the block can be flexibly specified, which can make full use of the memory capacity and flexibly adjust the number of batches for each block.

[0023] Preferably, in step S3, the sorted sentence pair blocks are grouped according to the total number of terms in each batch; the sentence pairs in each group are padded with zeros at the end according to the maximum sentence pair length in the group to obtain sentence pair groups with consistent lengths; and the sentence pair groups are further converted into batch tensors required by the training program to form dynamic-sized training batches.

[0024] Beneficial effects: The sentence pairs in the buffer block are sorted in ascending order according to the length of the sentence pairs in the memory, which can save a lot of memory. After sorting, the lengths of adjacent sentence pairs are close, so the lengths of sentence pairs in the same batch are close, so that less padding is required. When the total size of the batch is constant, the average number of sentence pairs in the batch can be increased, and the average padding of the batch can be reduced. Finally, a larger dynamic batch can be formed, thereby improving the training efficiency of machine translation (more sentence pairs can be trained per unit time) and training effect (large batches can make gradient estimation more accurate and gradient updates more effective and rapid).

[0025] To achieve the above-mentioned object, according to a second aspect of the present invention, there is provided a system for forming a dynamic-sized training batch for a neural machine translation system, the system comprising: a computer-readable storage medium and a processor;

[0026] The computer-readable storage medium is used to store executable instructions;

[0027] The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the dynamic size training batch formation method for the neural machine translation system described in the first aspect.

[0028] Preferably, the processor comprises a configuration module for maintaining the correspondence between parallel corpus type and parser, the correspondence between text language and segmenter, the dictionary size, the sentence pair block size and the batch size.

[0029] In general, the above technical solutions conceived by the present invention can achieve the following beneficial effects:

[0030] The present invention dynamically adjusts the training batch size according to the number of sentence entries in the neural machine translation system training. After sorting, the short sentence pairs in the sentence pair block are placed in front, and the sentence pairs with similar lengths are adjacent. Since the sentence pairs with similar number of entries are adjacent in the block after sorting, the sentence pairs in each group after grouping are relatively close in the number of entries, and the sentence pairs and the length of the filling are relatively small. In this way, a training batch with a larger average length can be formed under the same video memory, thereby improving the efficiency and effect of training. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A flow chart of a method for forming a dynamic size training batch for a neural machine translation system provided by an embodiment of the present invention;

[0032] Figure 2 A schematic diagram of the structure of a device for forming a dynamic size training batch for a neural machine translation system provided by an embodiment of the present invention;

[0033] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. In addition, the technical features involved in the various embodiments of the present invention described below can be combined with each other as long as they do not conflict with each other.

[0035] like Figure 1 As shown, the present invention provides a method for forming a dynamic size training batch for a neural machine translation system, comprising:

[0036] Step 100: Obtain sentence convolution from the parallel corpus.

[0037] Specifically, data is read from the parallel corpus one by one and parsed, the differences in the parallel corpus formats are shielded (different parallel corpus formats correspond to different specific subclassed sentence pair generators), and the source language sentences and target language sentence pairs in the parallel corpus are obtained in a streaming manner.

[0038] For example, no matter what format and structure the parallel corpus has, after parsing, this step generates an iterative stream of (source language sentences, target language sentences).

[0039] Step 101: clause convolution generates and saves source language and target dictionaries.

[0040] Specifically, the sentence pair stream is iterated to obtain sentence pairs, the source language and target language segmenters are obtained from the configuration, and the source language sentences and target language sentences in the sentence pairs are segmented by the segmenters to obtain the word lists of the source language sentences and the target language sentences. The unique words maintained in the source language and target language dictionaries are updated through the word lists. The unique words of the source language and target language dictionaries are numbered, and the dictionaries are saved for text vectorization.

[0041] For example, after the English-Chinese sentence pair (I am a teacher, 我是老师) is segmented by the English (word segmentation) and Chinese segmenters (character segmentation), the resulting lists of terms are I, am, a, teacher and 我,是,老,师 respectively. The source language dictionary and the target language dictionary are initially empty. After processing this sentence pair, their unique term sets are {I, am, a, teacher} and {我,是,老,师} respectively. Finally, the numbered dictionaries internally are a dictionary mapping structure: {a:1, the:2, …}, {是:1, 的:2, …}.

[0042] Step 102: Vectorize the source language and target language sentences in the parallel corpus.

[0043] Specifically, generate the sentence stream of the parallel corpus according to Step 100, load the source language and target language dictionaries, iterate over the sentence stream, obtain the source language and target language segmenters from the configuration, use the segmenters to segment the source language sentence and the target language sentence, and look up the dictionary to convert the segmented terms into numbered IDs, thereby vectorizing the text sentence into a sequence of digital IDs.

[0044] For example, after segmentation and conversion, the sentence pair (I am a teacher, 我是老师) generates a tuple of lists of digital sequences: ([81, 9, 1, 135], [11, 1, 99, 189]).

[0045] Step 103: Sort the sentence pairs in the vectorized sentence pair block in ascending order according to the number of terms in the sentence.

[0046] Specifically, buffer the vectorized sentence pairs in chunks. Each time, take a vectorized sentence pair block, and sort the sentence pairs in the vectorized sentence pair block (a tuple of two digital sequences) in ascending order according to the total length of the two digital sequences. The size of the buffered sentence pair block can be specified according to the configuration, but the sentence pairs in a buffered sentence pair block should be able to form multiple batches.

[0047] After sorting, the short sentence pairs in the sentence pair block are in the front, and the sentence pairs with similar lengths are adjacent.

[0048] Step 104: Group and pad the sorted sentence pair block to generate dynamically sized training batches.

[0049] Specifically, group the sorted sentence pair block according to the total number of terms in each batch specified in the configuration. Except for the last group, other groups have the same total number of terms. Pad the sentence pairs in each grouped group with 0 according to the longest (with the most number of terms) in the group to obtain dynamically sized training batches with the same total number of terms but different numbers of sentence pairs.

[0050] Since sentence pairs with similar number of terms are adjacent in the batch after sorting, the sentence pairs in each group after grouping are close in number of terms, so the number of sentence pairs and the length of the filling are small. In addition, the previous training batch has more sentence pairs, while the later training batch has fewer sentence pairs.

[0051] like Figure 2 As shown, it is a structural schematic diagram of a dynamic batch generation device of a neural machine translation system provided by an embodiment of the present invention, comprising: a sentence pair generation module 200, a preprocessing module 201, a dictionary module 202, a vectorization module 203, a cache module 204, a sorting and grouping module 205, a batch generation module 206 and a configuration module 207, wherein:

[0052] The sentence pair generation module reads and parses the parallel corpus to generate sentence pair streams.

[0053] Specifically, the sentence pair generation module reads data from the parallel corpus one by one, parses it, masks the format differences of the parallel corpus (different parallel corpus formats correspond to different specific subclassed sentence pair generators), and obtains the source language sentence and target language sentence pairs in the parallel corpus in a streaming manner. The sentence pair generation module is also an iterator and generator of source language sentence and target language sentence tuples on the parallel corpus.

[0054] The preprocessing module scans the parallel corpus and generates and saves the source language and target language dictionaries.

[0055] Specifically, the preprocessing module obtains sentence pairs from the sentence pair generation module, uses the configured source language and target language segmenters to obtain the entry lists of the source language text and the target language text, and constructs source language and target language dictionaries, that is, mappings of source language and target language entries to IDs, and saves the dictionaries for text vectorization.

[0056] Dictionary module, saves and loads dictionaries, and completes the mapping between entries and IDs.

[0057] Specifically, the dictionary module serializes and deserializes the dictionary structure, queries the corresponding ID according to the entry, and queries the text entry according to the ID.

[0058] The vectorization module vectorizes the sentence pairs to obtain vectorized sentence pairs.

[0059] Specifically, the vectorization module obtains sentence pairs from the sentence pair generation module, tokenizes the source language sentences and the target language sentences using the configured source language and target language segmenters, and maps the tokens to identification IDs using the dictionary generated by the preprocessing module, and finally generates a list array or irregular tensor for deep learning model training.

[0060] The cache module caches the vectorized sentence pairs.

[0061] Specifically, the cache module caches the vectorized sentence pairs generated by the vectorization module and provides reading of the vectorized sentence pairs in a block manner, and the size of the block can be configured according to the available memory.

[0062] The sorting and grouping module sorts and groups the vectorized sentence blocks.

[0063] Specifically, the sorting and grouping module sorts each vectorized sentence pair block obtained from the cache module in ascending order according to the length of the vectorized sentence pair (the length of the sentence pair is the sum of the number of source language sentence entries and the number of target language sentence entries), and groups the blocks according to the set batch size (the batch size is the sum of the lengths of all sentence pairs in a batch), and except for the last group, the length of each group is equal to the set batch size.

[0064] Batch generation module, generates dynamically sized training batches from sorted groups.

[0065] Specifically, the batch generation module pads the sentence pairs of different lengths in a group of data generated by the sorting and grouping module with zeros at the end according to the maximum sentence pair length in the batch to obtain sentence pair groups with consistent lengths, and further converts the sentence pair groups into batch tensors required by the training program.

[0066] The configuration module is used to maintain customizable configuration information during operation.

[0067] Specifically, the configuration module maintains configuration information such as the correspondence between parallel corpus type and corpus parser, the correspondence between text language and segmenter, dictionary size, sentence pair block size, global batch size, etc., and accesses the configuration information through attribute access.

[0068] The training batches generated by the method and device for forming a dynamic-size training batch for a neural machine translation system provided by the present invention can include more sentence pairs on average, that is, the average batch size is larger, which can reduce the training convergence time of the neural machine translation system, improve training efficiency, better approach the global optimum, and obtain better training effects; in addition, the training batches generated according to the present invention have smaller padding, which can make fuller use of the computing and storage capabilities of the GPU, reduce the training time of the entire corpus, and improve the training speed.

[0069] Figure 3 A schematic diagram of the physical structure of an electronic device provided by an embodiment of the present invention, such as Figure 3As shown, the electronic device may include: a processor 310, a communication interface 320, a memory 330 and a communication bus 340, wherein the processor 310, the communication interface 320 and the memory 330 communicate with each other through the communication bus 340. The processor 310 may call a computer program stored in the memory 330 and executable on the processor 310 to execute the dynamic training batch formation method for neural machine translation provided by the above-mentioned method embodiments.

[0070] In addition, the logic instructions in the above-mentioned memory 330 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the embodiment of the present invention is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0071] An embodiment of the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the method for forming a dynamic training batch for neural machine translation provided by the above-mentioned method embodiments is implemented.

[0072] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0073] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0074] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for forming dynamically sized training batches for a neural machine translation system, characterized in that: The method includes: S1. Parse the parallel corpus, segment and vectorize the corpus, and obtain vectorized sentence pairs; S2. Cache the vectorized sentence pairs in blocks, take one vectorized sentence pair block each time, and sort the sentence pairs in the vectorized sentence pair block in ascending order according to the total length of the two digital sequences; S3. Group the sorted sentence pair blocks according to the total number of terms in each batch; pad the sentence pairs in each group with zeros according to the maximum sentence pair length in the group to obtain sentence pair groups with consistent lengths; further convert the sentence pair groups into batch tensors required by the training program to form dynamic-sized training batches.

2. The method according to claim 1, characterized in that Step S1 includes: S11. Parse the parallel corpus to obtain source language and target language sentence convection; S12. segment the sentences in the sentence stream, generate source language and target language dictionaries and save them; S13. Use the generated source language and target language dictionaries to vectorize the parallel corpus.

3. The method according to claim 2, characterized in that In step S11, parallel corpora in different formats use different parsers, and the correspondence between the corpus formats and the parsers is stored in a configuration file.

4. The method according to claim 2, characterized in that In step S12, different text languages ​​use different segmenters, and the correspondence between the text languages ​​and the segmenters is stored in a configuration file.

5. The method according to claim 2, characterized in that In step S13, if the size of the generated dictionary is specified, a counting structure of the entries and the number of times they appear in the text is maintained, the number of occurrences is updated during the scanning of the text, and the counting structure is sorted in descending order according to the number of occurrences after the scanning is completed, and the entries of the previously specified dictionary size in the sorted counting structure are used as dictionary content.

6. A system for dynamically sized training batches for a neural machine translation system, characterized in that: The system includes: a computer-readable storage medium and a processor; The computer-readable storage medium is used to store executable instructions; The processor is used to read the executable instructions stored in the computer-readable storage medium to execute the dynamic size training batch formation method for the neural machine translation system according to any one of claims 1 to 5.

7. The system according to claim 6, characterized in that The processor includes a configuration module for maintaining a correspondence between parallel corpus types and parsers, a correspondence between text languages ​​and segmenters, a dictionary size, a sentence pair block size, and a batch size.

Images