Chinese-English conditional tense machine translation method based on pre-training model and data enhancement technology
Through the self-attention mechanism and Chinese-BERT classifier combined with data enhancement algorithm, a conditional tense translator was constructed, which solved the problem of poor conditional tense translation effect of traditional machine translation models and improved the accuracy of tense translation from Chinese to English sex.
Patent Information
- Application Number
- CN202310580943.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-22
- Publication Date
- 2024-11-22
AI Technical Summary
The traditional "Chinese-English" machine translation model has poor performance in conditional tense translation, especially due to the complexity of Chinese conditional tense and sample scarcity, which leads to inaccurate translation.
The self-attention mechanism is used to encode the context tense information, and the conditional tense classifier is constructed based on the pre-trained model Chinese-BERT, and the conditional tense and non-conditional tense data sets are constructed through the data augmentation algorithm, combining with the machine translation model HAN- NMT builds conditional tense translators and non-conditional tense translators, and judges and decides to use the appropriate tense translators for translation.
Effectively capture the context tense information of Chinese sentences and English documents, improving the accuracy and effectiveness of "Chinese-English" conditional tense translation.
Smart Images

Figure CN119005215A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of artificial intelligence in computer technology, especially the field of natural language processing and machine translation in artificial intelligence. It adopts the pre-trained model Chinese-BERT and a data enhancement algorithm to construct a "Chinese-English" machine translation system. The system captures the temporal information of the context through the self-attention mechanism, which can effectively improve the translation effect of the machine translation model in the conditional tense during "Chinese-English" translation. Background Art
[0002] Translation is a process of converting one thing into another, usually for sequence-to-sequence conversion. The translation problem that the field of Natural Language Processing (NLP) mainly focuses on is the translation between natural languages. Translation software can automatically convert a section of Chinese input by the user into a corresponding English expression. The technology of using a computer to translate natural languages like this is called Machine Translation (MT), in which the language to be translated is called the source language, and the language corresponding to the translation result is called the target language. Unless otherwise specified, the "translation" mentioned in the following content of this manual refers to the translation between natural languages.
[0003] Tense information is an important component of natural language. In grammar, tense is used to indicate the time when something happens, usually reflected by various forms of verbs.
[0004] The conditional tense in English is a special tense. While it's used far less frequently than the present, past, and future tenses, it plays a crucial role. The conditional tense often appears with the subjunctive mood, which expresses subjective wishes, suspicions, suggestions, or assumptions that contradict reality rather than objective facts. Sentences in the conditional tense mostly use modal verbs, including "could," "should," "would," and "might."
[0005] Unlike the present, past, and future tenses, which can be expressed using auxiliary words related to time concepts, the conditional tense in Chinese uses words related to wishes, suspicions, suggestions, or assumptions to express a subjunctive mood. However, it is often difficult to determine whether a Chinese sentence should use the conditional tense, even if it contains words related to the subjunctive mood. Therefore, to correctly determine whether a Chinese sentence uses the conditional tense when translated into English, it is necessary to fully understand the context.
[0006] The Transformer neural network model can effectively encode contextual information in text using the attention mechanism, and the pre-trained models derived from it have even stronger information representation capabilities. With the rapid development of artificial intelligence technology in recent years, using machines to replace humans in translation has gradually become a reality. However, some excellent translation models often overlook the translation effect of tense, especially tenses such as conditional tenses, which are "rarely used but very important." Summary of the Invention
[0007] The present invention aims to solve the problem that the traditional "Chinese-English" machine translation model has poor translation effect in conditional tenses. The technical solution is:
[0008] 1) Encode the temporal information of the context based on the self-attention mechanism.
[0009] 2) Build a conditional temporal classifier based on the pre-trained model Chinese-BERT.
[0010] 3) Based on the data enhancement algorithm, conditional temporal datasets and unconditional temporal datasets are constructed.
[0011] 4) Based on the machine translation model HAN-NMT and the conditional temporal dataset and unconditional temporal dataset, a conditional temporal translator and an unconditional temporal translator are constructed.
[0012] The beneficial effects of the present invention are as follows: the "Chinese-English" conditional tense machine translation system can effectively capture the features in Chinese sentences and the contextual tense information in English documents, and then determine whether a Chinese sentence should use the conditional tense when translated into English. Then, based on the result of the judgment, it is decided whether to use a conditional tense translator or a non-conditional tense translator to perform the translation, thereby ultimately improving the translation effect of the translation system in terms of conditional tense. BRIEF DESCRIPTION OF THE DRAWINGS
[0013] Figure 1 Schematic diagram of the structure of the conditional temporal classifier.
[0014] Figure 2 This is the conditional temporal enhancement algorithm used in the present invention.
[0015] Figure 3 This is the overall structural diagram of the "Chinese-English" conditional tense translation system implemented in the present invention. DETAILED DESCRIPTION
[0016] In order to better understand the technical content of the present invention, specific embodiments are given below in conjunction with the accompanying drawings.
[0017] The present invention involves the following key technologies: using the self-attention mechanism to encode the contextual temporal information of English, building a conditional temporal classifier using the pre-trained model Chinese-BERT, designing a conditional temporal data enhancement algorithm, and building a conditional temporal translator and an unconditional temporal translator based on the paragraph-level machine translation model HAN-NMT.
[0018] 1. Using the self-attention mechanism to encode English contextual temporal information
[0019] The self-attention mechanism is a global encoding method characterized by its ability to incorporate global contextual information. Broadly speaking, the attention mechanism is a means of capturing contextual information in a serialized representation. We note that from the perspective of tense, the conditional tense in English may have a certain relationship with the tense of the context. For example, the conditional tense is often used somewhere in the narrative of a fact. This fact typically has some tense characteristics, usually consistency characteristics, that is, sentences using the conditional tense often have the same or similar tense in the context, such as present or past tense.
[0020] Therefore, we make the following assumption: sentences using conditional tenses have certain temporal characteristics in their context. The attention mechanism can automatically help us capture the contextual relationships of a sequence. Therefore, we propose a method based on the self-attention mechanism to introduce contextual temporal information. The specific approach is as follows:
[0021] For the i-th Chinese sentence in the document Define its contextual temporal information for:
[0022]
[0023] Among them, ctxSize is an integer variable used to define the range size of context information, and the default value is 3. For sentences The temporal information of the first ctxSize sentences and the last ctxSize sentences; the function SelfAttention is used to calculate The self-attention encoding results are as follows. Their specific definitions are:
[0024]
[0025]
[0026] Where W q 、W k 、W v are three linear transformation matrices, The role of is to prevent the softmax distribution from being too steep. i For sentences The temporal information of is a four-dimensional vector composed of 0 and 1. We assume that pres i 、past i 、fut i and cond i There are four 0-1 variables, representing the present tense, past tense, future tense and conditional tense respectively. When any of the above tenses are used, the corresponding 0-1 variable is set to 1, otherwise it is set to 0.
[0027] t i =(pres i , past i , fut i ,cond i )
[0028] 2. Conditional temporal classifier based on Chinese-BERT
[0029] The conditional temporal classifier we designed is as follows Figure 1 As shown in the figure, the algorithm consists of three main parts: 1) a Chinese pre-trained model, Chinese-BERT, which is fine-tuned on the dataset constructed in the previous section. 2) the contextual temporal information encoding module, TenseAttention, designed above. 3) a linear classification neural network, which receives the output of the pre-trained model, concatenates the [CLS] vector and the output of TenseAttention as the classification input, and outputs a binary classification result.
[0030] We use the following set of formulas to formally define the entire process of the conditional temporal classifier:
[0031] set up is the i-th Chinese sentence to be classified, and its hidden layer representation on the pre-trained model Chinese-BERT is:
[0032]
[0033] We take The first vector of is used as the input of the linear neural network, denoted as
[0034]
[0035] Then it will and The two layers are connected and input to the linear classification layer Linear, whose output is a two-dimensional vector Outputi :
[0036]
[0037] Then use the softmax function to i Perform normalization to obtain the final classification score Score i :
[0038]
[0039] at this time and It can be regarded as a sentence The probability of being classified as a positive sample / negative sample, so take and The classification label corresponding to the larger value in The classification results.
[0040] We use cross entropy as the loss function, which is calculated using the following formula: For sentences The true classification label of the positive sample is 1 and the negative sample is 0:
[0041]
[0042] 3. Conditional Temporal Data Augmentation Algorithm
[0043] Our conditional tense augmentation algorithm consists of two parts: one converts non-conditional tense sentences into conditional tense sentences; the other performs the opposite, converting conditional tense sentences into non-conditional tense sentences. These two parts augment the IWSLT14 dataset into two datasets: the conditional tense augmentation dataset and the non-conditional tense augmentation dataset. Below we describe the augmentation algorithms for these two datasets.
[0044] The definition and conversion algorithm of the conditional temporal enhancement dataset CondData are as follows, where 0≤i≤N-1, and N is the number of "Chinese-English" parallel corpus data contained in the IWSLT14 dataset:
[0045]
[0046] Figure 2 The specific algorithm steps for constructing the CondData dataset are shown and described as follows:
[0047] 1) For any Chinese-English sentence pair in the IWSLT14 dataset, record it as
[0048] 2) Judgment Have you used the conditional tense? If so, skip to step 7.
[0049] 3) Judgment Does it contain modal verbs can / will / shall / may? If not, skip to step 5.
[0050] 4) The modal verbs in are replaced according to the following rules: can→could, will→would, shall→should, may→might.
[0051] 5) Use NLTK tools to Perform word segmentation and part-of-speech tagging to obtain The verb set VERB in .
[0052] 6) Randomly select a non-auxiliary verb (do / does / doing / done / did) from VERB and use NLTK tools to obtain the verb prototype of the verb Randomly select a modal verb modal from could / would / should / might, and The verb in is replaced by
[0053] 7) Remember this time for Match the sentences Add to the dataset CondData.
[0054] The definition of the non-conditional temporal enhancement dataset NoCondData is as follows, where 0≤i≤N-1, and N is the number of "Chinese-English" parallel corpus data contained in the IWSLTl4 dataset:
[0055]
[0056] Compared to CondData, NoCondData is much simpler to construct. All that's required is replacing the modal verbs could / would / should / might in all English sentences in IWSLTl4 with can / will / shall / may. However, one detail that requires extra attention is that since English sentences contain "d" like "I'd like to go shopping," where "d" is actually an abbreviation of "would," all "d"s need to be replaced with "will."
[0057] 4. Conditional Tense Translator & Unconditional Tense Translator
[0058] Based on the two datasets obtained in 3. - the conditional temporal enhancement dataset and the non-conditional temporal enhancement dataset, we use the paragraph-level translation model HAN-NMT to train on them respectively, and obtain two translation models - the conditional temporal translator and the non-conditional temporal translator, denoted as CondTranslator and NoCondTranslator respectively.
[0059] To sum up, the overall structure of our final "Chinese-English" conditional temporal machine translation system is as follows Figure 3 As shown in Figure 2, the model mainly consists of three parts: 1) the conditional temporal classifier proposed in 2., which introduces the contextual temporal information proposed in 1., denoted as TenseClassifierWithCtx, 2) the conditional temporal translator CondTranslator and the non-conditional temporal translator NoCondTranslator proposed in 4., 3) the fusion module Fusioner, which is used to integrate the translation results of CondTranslator and NoCondTranslator.
[0060] The processing steps of the entire solution are as follows:
[0061] 1) For each Chinese sentence in a Chinese document First input it to the conditional temporal classifier TenseClassifier, and let the output of the classifier be X.
[0062] 2) If X=True, then Input to the conditional tense translator CondTranslator to get the English sentence using conditional tense
[0063] 3) If X=False, then Input to the non-conditional tense translator NoCondTranslator to get an English sentence that does not use conditional tenses
[0064] 4) Fusioner module integrates the output of step 2) or 3) as Added to the English document of the final translation result.
[0065] It should be noted that to speed up the calculation, before all steps begin, the TenseAttention layer in TenseClassiferWithCtx will perform a complete scan of the document from beginning to end, use the TMV-annotator tool to calculate the contextual temporal information TenseCtx of each Chinese sentence in the test set, and cache it in TenseCtxCache. In this way, when the subsequent data stream passes through TenseAttention, the model does not need to call TMV-annotator for temporal analysis and calculation every time. It only needs to read the corresponding TenseCtx from the cache TenseCtxCache according to index i.
Claims
1. A "Chinese-English" conditional temporal machine translation method based on a pre-trained model and data augmentation technology, characterized in that The traditional Chinese-English machine translation method is subdivided into three major processes: classification-translation-fusion using the pre-trained model Chinese-BERT and the conditional temporal data augmentation algorithm. In addition, the contextual temporal information based on the self-attention mechanism is additionally introduced to improve the translation effect of conditional tense in Chinese-English machine translation.
2. The method according to claim 1, characterized in that First, the IWSLT14 "Chinese-English" parallel corpus dataset is preprocessed to convert the XML format data into a "Chinese-English" sentence pair dataset. Then, the English automatic tense annotation tool TMV-annotator is used to annotate the English in the dataset obtained from 2, and all verb phrases and their corresponding tense categories in each English sentence are obtained. The "Chinese-English" sentence pairs that use conditional tenses are selected as positive samples, and the same number of negative samples are selected to construct a "Chinese-English" conditional tense binary classification dataset.
3. Introducing contextual temporal information using the self-attention mechanism. The present invention considers that a sentence in English that uses conditional tense also has certain characteristics in terms of tense in its context, so the present invention is characterized in that the self-attention mechanism is used to construct an encoding of contextual temporal information for an English document, and the contextual temporal information features of each sentence are obtained.
4. Use the pre-trained model Chinese-BERT, the self-attention mechanism and a linear classification neural network to build a conditional tense classifier. The classifier is fine-tuned on the conditional tense binary classification dataset obtained in 2. The present invention utilizes the temporal features of the context in 3, connects it with the [CLS] vector output by Chinese-BERT, and inputs them together to the linear classification neural network for label prediction. The conditional tense classifier can capture features related to English conditional tenses in Chinese sentences. It accepts a Chinese sentence as input and determines whether the sentence should use conditional tenses when translated into English.
5. Claim 5 is characterized by the design of a conditional tense data enhancement algorithm. The method of the present invention utilizes the natural language toolkit NLTK to perform sentence segmentation, word segmentation, and part-of-speech tagging on the data set obtained from claim 2. The conditional tense enhancement algorithm can effectively 1) convert an English sentence that does not use conditional tense into a sentence that uses conditional tense without changing its grammatical structure and semantics; correspondingly, it can also 2) "de-conditionalize" an English sentence that uses conditional tense. We use processing 1) and 2) to perform data enhancement on the data set obtained from claim 2, respectively, and obtain two corresponding data sets: a conditional tense enhanced data set and a non-conditional tense enhanced data set.
6. Use the neural machine translation model HAN-NMT to train on the two datasets obtained in 5, the conditional tense enhancement dataset and the unconditional tense enhancement dataset, to obtain the conditional tense translator and the unconditional tense translator. The conditional tense translator always translates a Chinese sentence into an English sentence using conditional tense, while the unconditional tense translator does the opposite.
7. The feature of claim 7 is that a "Chinese-English" conditional tense machine translation model is constructed using the conditional tense classifier in 4 and the conditional tense translator and non-conditional tense translator in 6. The model is divided into three parts: 1) Classification. The input Chinese sentence is first submitted to the conditional tense classifier to determine whether the conditional tense should be used when translating it into an English sentence. 2) Translation. If the output of 1) is true, the input is submitted to the conditional tense translator for translation. If the output of 1) is false, it is submitted to the non-conditional tense translator for translation. 3) Fusion. The outputs in 2) are fused to obtain the final English document.
