Semantic consistency text-level neural machine translation method based on topic fusion
Through topic fusion technology, including topic word extraction and hybrid expert model, the problem of insufficient topic drift and semantic consistency in chapter-level translation of neural machine translation models is solved, achieving higher quality and efficient translation effects.
Patent Information
- Application Number
- CN202411148012.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-11-29
AI Technical Summary
Traditional neural machine translation models are prone to topic drift when translating long documents or chapter levels, resulting in poor logical coherence and semantic consistency of translation results, and insufficient robustness of the model, making it difficult to deal with multi-topic documents.
Using semantic consistency chapter-level neural machine translation method based on topic fusion, the topic word extraction, Adapter module and hybrid expert model MoE can achieve effective fusion of topic information migration and source language information, and using topic word soft prompts and semantic consistency Sexual loss function optimization model training.
It improves the quality and efficiency of chapter-level translation, avoids topic drift, enhances the logical coherence and semantic consistency of translation results, and significantly improves the robustness of the model.
Smart Images

Figure CN119047492A_ABST
Abstract
Description
Technical Field
[0001] The invention relates to a semantically consistent chapter-level neural machine translation method based on topic fusion, and belongs to the field of natural language processing. Background Art
[0002] In the development of modern machine translation technology, Neural Machine Translation (NMT) has gradually become mainstream due to its end-to-end training method and strong semantic capture capabilities. However, traditional NMT models often face the phenomenon of topic drift when facing long documents or paragraph-level translation tasks, resulting in poor performance of translation results in terms of logical coherence and semantic consistency between sentences. This topic drift phenomenon occurs mainly because the NMT model does not capture enough global semantic information of the text during the translation process. Especially when dealing with texts involving multiple topic conversions, the model is difficult to effectively maintain the consistency of the topic, which in turn affects the accuracy and consistency of the translation.
[0003] In recent years, researchers have proposed a variety of improvement methods, such as enhancing the semantic capture ability of the model through contextual information, or reducing the semantic deviation between sentences through specific strategies. However, the existing methods still have many shortcomings in practical applications, which are mainly manifested in the following points: Insufficient capture of topic information: Most existing models mainly rely on local contextual information and cannot fully utilize the global topic information at the chapter level. This causes the model to be prone to topic incoherence when facing complex texts, affecting the translation quality; Insufficient global semantic consistency: Due to the lack of effective modeling of global topics, existing NMT models are difficult to ensure the logical coherence between sentences, especially in the translation of long documents. The semantic consistency of the translation results is poor, and topic drift is prone to occur; The model is not robust: When dealing with different topic information, existing models often lack sufficient flexibility and cannot dynamically adjust the translation strategy according to different topic information, resulting in poor performance in multi-topic document translation. Summary of the invention
[0004] The present invention proposes a semantically consistent chapter-level neural machine translation method based on topic fusion to solve the technical problems in the above-mentioned technologies, realizes the migration of topic information and the effective fusion of topic information and source language information, and improves the quality and efficiency of chapter-level translation.
[0005] The technical solution of the present invention is: a semantic consistency chapter-level neural machine translation method based on topic fusion, comprising the following steps:
[0006] Step 1. Use the keyword extraction tool to extract key keywords from each chapter;
[0007] Step 2: Load the results of sentence-level training to the chapter-level, and perform chapter-level machine translation training on the chapter-level data;
[0008] Step 3: Introduce the topic migration mechanism and use Adapter to achieve the migration of sentence-level information to chapter-level information, and further optimize the chapter-level data;
[0009] Step 4: Apply the semantic consistency loss function guided by keywords to optimize the model training process.
[0010] Furthermore, the step 1 includes: extracting keywords from the chapter-level data using a latent Dirichlet allocation (LDA) model.
[0011] Furthermore, the specific steps of extracting keywords from the chapter-level data using the latent Dirichlet allocation LDA model are as follows:
[0012] Step 1.1, preprocess each chapter-level dataset to meet the input requirements of the LDA model, including removing stop words and low-frequency words;
[0013] Step 1.2, input the preprocessed text data into the LDA model, set the number of topics T, and the model outputs the topic distribution of each chapter;
[0014] Step 1.3: Select several high-probability words from the topic distribution of each chapter as key words, and associate these key words with the corresponding chapter data for subsequent translation model training and topic transfer.
[0015] Furthermore, the specific steps of Step 2 are as follows:
[0016] Step 2.1, load the trained sentence-level model parameters into the paragraph-level translation model to initialize the weights and configurations of the paragraph-level model;
[0017] Step 2.2, input the chapter-level data and extracted keywords into the chapter-level translation model with loaded parameters, and further optimize the model parameters through training, so that the model can improve the consistency and accuracy of translation based on the use of contextual information and topic information.
[0018] Furthermore, the step Step3 includes: adopting a prompt learning strategy guided by topic word soft prompts, in which each soft prompt is regarded as an "expert", and a hybrid expert model MoE is formed by the fusion of multiple soft prompts; in addition, by using the BERT model to encode the topic words, generating high-quality word vector representations, and introducing a topic word-aware multi-level dynamic fusion mechanism, these topic information are effectively integrated with the source language information, thereby achieving a topic migration effect.
[0019] Furthermore, the specific steps of Step 3 are as follows:
[0020] Step 3.1, after obtaining the encoding representation H of the source language sentence m After obtaining the vector representation E of the topic word and the topic word, a prompt learning strategy guided by the topic word soft prompt is adopted to fuse the topic word information with the source language information; specifically, the information fusion is performed through the hybrid expert model MoE; first, the topic word vector representation is average pooled:
[0021]
[0022] Among them, E mean is the average representation of the subject words, T is the number of subject words, is the vector representation of the i-th topic word.
[0023] Next, the MoE model is used to fuse the outputs of multiple “experts”; each “expert” corresponds to a specific topic representation, and the MoE model obtains the weight α of each expert through learning u , these expert outputs are fused into the final topic representation:
[0024]
[0025] Among them, α u It is the weight obtained through model learning, which represents the contribution of each expert to the final fusion result.
[0026] In getting the keyword expression h MoE and the source language encoding H m Finally, the two types of information are further integrated through the gating mechanism; first, the gating value G is calculated:
[0027] G=σ(W g ·[H m ,h MoE ])
[0028] Among them, σ represents the Sigmoid activation function, W g is the weight matrix of the fully connected layer, [H m ,h MoE ] represents the input vector obtained by performing element-by-element operations on the two;
[0029] Finally, the gating value is used to represent the topic word h MoE and the source language encoding H m After fusion weighting, the final fusion expression is as follows:
[0030] H fused =G⊙Hm +(1-G)⊙h MoE
[0031] Among them, ⊙ represents element-by-element multiplication;
[0032] Step 3.2, introduce the Adapter module. The Adapter module is a lightweight bottleneck structure. By introducing the Adapter in the Transformer layer, it is used to capture and utilize the topic information. The process of the Adapter module is as follows:
[0033] H adapted =Adapter(H fused )
[0034] Through the Adapter module, the model can better adapt to the topic information at different levels.
[0035] Furthermore, the step 4 includes the following:
[0036] The consistency loss function is introduced, which aims to maintain the dominance of the source language information while effectively utilizing the information of the topic words. Specifically, the consistency loss is expressed as:
[0037]
[0038] Among them, λ is a weight coefficient used to balance the contribution of source language information and topic information.
[0039] The beneficial effects and advantages of the present invention are:
[0040] 1. The present invention proposes a hint learning strategy guided by soft hints of subject words. This method performs soft hint processing on each subject word and regards it as an "expert" to form a Mixture-of-Experts (MoE) model to effectively capture the global semantic information of the text and avoid topic drift. In addition, a semantic consistency loss function guided by subject words is introduced to balance the contribution of source language information and topic information, avoid the model's over-reliance on subject words, and thus improve the consistency and accuracy of the translation results.
[0041] 2. The present invention proposes a BERT model encoding and multi-level dynamic fusion mechanism. This method encodes the subject words through the BERT model to generate high-quality word vector representations, and effectively fuses these subject information with the source language information through a multi-level dynamic fusion mechanism to ensure the accurate transmission of subject information and the consistency of global semantics.
[0042] 3. The method of the present invention has been extensively experimentally verified on multiple benchmark datasets, and in-depth analysis has further demonstrated the effectiveness of the paragraph-level neural machine translation module proposed in the present invention. BRIEF DESCRIPTION OF THE DRAWINGS
[0043] Figure 1 It is a flow chart of the present invention;
[0044] Figure 2 This is a schematic diagram of the specific structure of the semantically consistent chapter-level neural machine translation model based on topic fusion proposed in the present invention. DETAILED DESCRIPTION
[0045] The present invention will be further described below in conjunction with the accompanying drawings.
[0046] Embodiment 1, as Figure 1-2 As shown, the present invention proposes a semantic consistency chapter-level neural machine translation method based on topic fusion, and the technical solution adopted is:
[0047] Step 1. Use the keyword extraction tool to extract key keywords from each chapter;
[0048] This paper studies the English-German machine translation task and selects three commonly used datasets for evaluation: TED Talks (IWSLT2017), News Commentary (NC2016), and European Conference (Europarl v7). The datasets used are all public datasets, and their sources are public and legal, so there is no need to worry about infringement or intellectual property rights. We strictly abide by the use agreement and terms of the data provider to ensure the legal acquisition and use of the data.
[0049] As a further illustration of the present invention, the data is first preprocessed using a tool to obtain sentence-level data and paragraph-level data;
[0050] Among them, the data is preprocessed using tools to obtain sentence-level data and chapter-level data, and the key words are extracted from each chapter using a keyword extraction tool. The above process can be carried out in the following way:
[0051] (1) Data segmentation and TrueCasing processing: Use the Moses toolkit to perform segmentation on all data sets and divide the original text into sequences of words or subwords;
[0052] (2) Perform TrueCasing, which is to adjust the capitalization of words according to their frequency and context in the corpus to ensure the consistency and accuracy of the text;
[0053] (3) Perform byte pair encoding (BPE) preprocessing on the text to obtain a vocabulary and text data after subword segmentation;
[0054] (4) The chapter-level data is segmented based on the maximum number of tags, and the data is preprocessed into sentence-level data and chapter-level data for storage. <s>< / s> Indicates the beginning and end of each sentence.
[0055] (5) Use the Latent Dirichlet Allocation (LDA) model to extract keywords from the chapter-level data.
[0056] Furthermore, the specific steps of extracting keywords from the chapter-level data using the latent Dirichlet allocation LDA model are as follows:
[0057] Step 1.1, preprocess each chapter-level dataset to meet the input requirements of the LDA model, including removing stop words and low-frequency words;
[0058] Step 1.2, input the preprocessed text data into the LDA model, set the number of topics T, and the model outputs the topic distribution of each chapter;
[0059] Step 1.3: Select several high-probability words from the topic distribution of each chapter as key words, and associate these key words with the corresponding chapter data for subsequent translation model training and topic transfer.
[0060] As a further illustration of the present invention, sentence-level data is then subjected to sentence-level machine translation training;
[0061] As a further illustration of the present invention, the specific method of performing sentence-level machine translation training on sentence-level data can be performed in the following manner:
[0062] (1) Given a source document X = (X 1 ,…,X m ,…,X M ), which contains M sentences, each sentence Contains I words. The corresponding target document is Y = (Y 1 ,…,Y m ,…,Y M ), which contains M sentences, each sentence Contains J words. The topic word we introduced is W = {w 1 ,w 2 ,…,w T It is composed of T keywords to enhance the model’s understanding of global semantics.
[0063] (2) Use the word embedding layer to extract sentence-level representations as follows
[0064]
[0065] Among them, Emb is the source sentence embedding layer with position embedding;
[0066] (3) Use several stacked Transformers to take the source sentence as input to capture sentence-level semantics. The representation process is as follows:
[0067]
[0068] Among them, sTrans s Represents a sentence-level stacked Transformer encoder with multi-head attention blocks and a feed-forward network;
[0069] (4) Train sentence-level models for sentence-level representations using H m As a low-level local sentence representation. In the initial stage of the model, by using H m , the model is able to obtain sentence-level information and encode it into vector form.
[0070] (5) In order to better utilize the topic information in the text in text machine translation, we introduced the BERT (Bidirectional Encoder Representations from Transformers) model to encode the topic words, as shown below:
[0071] E=BERT(w)
[0072] Among them, w represents the input topic word sequence, and E represents the word vector matrix.
[0073] Step 2: Load the results of sentence-level training to the chapter-level, and perform chapter-level machine translation training on the chapter-level data;
[0074] Furthermore, the specific steps of Step 2 are as follows:
[0075] Step 2.1, load the trained sentence-level model parameters into the paragraph-level translation model to initialize the weights and configurations of the paragraph-level model;
[0076] Step 2.2, input the chapter-level data and extracted keywords into the chapter-level translation model with loaded parameters, and further optimize the model parameters through training, so that the model can improve the consistency and accuracy of translation based on the use of contextual information and topic information.
[0077] Step 3: Introduce the topic migration mechanism and use Adapter to achieve the migration of sentence-level information to chapter-level information, and further optimize the chapter-level data;
[0078] Furthermore, the step Step3 includes: adopting a prompt learning strategy guided by topic word soft prompts, in which each soft prompt is regarded as an "expert", and a hybrid expert model MoE is formed by the fusion of multiple soft prompts; in addition, by using the BERT model to encode the topic words, generating high-quality word vector representations, and introducing a topic word-aware multi-level dynamic fusion mechanism, these topic information are effectively integrated with the source language information, thereby achieving a topic migration effect.
[0079] Furthermore, the specific steps of Step 3 are as follows:
[0080] Step 3.1, after obtaining the encoding representation H of the source language sentence m After obtaining the vector representation E of the topic word and the topic word, a prompt learning strategy guided by the topic word soft prompt is adopted to fuse the topic word information with the source language information; specifically, the information fusion is performed through the hybrid expert model MoE; first, the topic word vector representation is average pooled:
[0081]
[0082] Among them, E mean is the average representation of the subject words, T is the number of subject words, is the vector representation of the i-th topic word.
[0083] Next, the MoE model is used to fuse the outputs of multiple “experts”; each “expert” corresponds to a specific topic representation, and the MoE model obtains the weight α of each expert through learning u , these expert outputs are fused into the final topic representation:
[0084]
[0085] Among them, α u It is the weight obtained through model learning, which represents the contribution of each expert to the final fusion result.
[0086] In getting the keyword expression h MoE and the source language encoding H m Finally, the two types of information are further integrated through the gating mechanism; first, the gating value G is calculated:
[0087] G=σ(W g ·[H m ,h MoE ])
[0088] Among them, σ represents the Sigmoid activation function, W g is the weight matrix of the fully connected layer, [H m ,h MoE ] represents the input vector obtained by performing element-by-element operations on the two;
[0089] Finally, the gating value is used to represent the topic word h MoE and the source language encoding H m After fusion weighting, the final fusion expression is as follows:
[0090] H fused =G⊙H m +(1-G)⊙h MoE
[0091] Among them, ⊙ represents element-by-element multiplication;
[0092] Step 3.2, introduce the Adapter module. The Adapter module is a lightweight bottleneck structure. By introducing the Adapter in the Transformer layer, it is used to capture and utilize the topic information. The process of the Adapter module is as follows:
[0093] H adapted =Adapter(H fused )
[0094] Through the Adapter module, the model can better adapt to the topic information at different levels.
[0095] Step 4: Apply the semantic consistency loss function guided by keywords to optimize the model training process.
[0096] Furthermore, the step 4 includes the following:
[0097] In order to prevent the model from over-relying on topic words during the translation process, we introduce a consistency loss function. This loss function aims to maintain the dominance of the source language information while effectively utilizing the information of the topic words. Specifically, the consistency loss can be expressed as:
[0098]
[0099] Among them, λ is a weight coefficient used to balance the contribution of source language information and topic information.
[0100] In order to prove the effectiveness of the proposed method, a large number of experiments were conducted on three datasets: TED, NC2016 and Europarl v7. The details of the three datasets are shown in Table 1. In order to accurately evaluate the performance of the method, s-BLUE and d-BLUE are used as evaluation indicators. These two indicators can objectively and accurately reflect the effect of machine translation. Two groups of experiments were set up. The first group is the main experimental results, which are compared with some previous works in this field. The second group of experiments is ablation experiments to verify the effectiveness of each module of the proposed method.
[0101] Table 1 Dataset details
[0102]
[0103] (1) Main experimental results
[0104] Table 2 Main experimental results
[0105]
[0106] The proposed method is compared with several recent paragraph-level neural machine translation methods. These comparable baselines include two main types of paragraph-level neural machine translation methods: Sent-level paragraph neural machine translation and Doc-level paragraph neural machine translation methods. Table 2 shows the main translation results of the model corresponding to the proposed method and other SOTA models on three datasets. The experimental results show that:
[0107] On the sentence-level and paragraph-level baselines, the model corresponding to the proposed method achieves higher BLEU scores on all datasets;
[0108] The proposed model achieved good results in the English → German translation task. Compared with the existing paragraph-level neural machine translation methods, the proposed model outperformed the existing SOTA models and improved the BLEU scores of most test sets. This confirms that the proposed method can effectively use semantic transfer to improve machine translation;
[0109] Compared with sentence neural machine translation: The paragraph-level neural machine translation model of the present invention significantly outperforms sentence neural machine translation, and has achieved a score improvement of 2 to 4 points on all test sets. The experimental results show that contextual information is effective in improving the quality of machine translation.
[0110] (2) Ablation experiment
[0111] To demonstrate the effectiveness of the proposed method, an ablation study was conducted to examine the impact of the proposed model on the performance of the TED and News datasets. The experimental results are shown in Table 3:
[0112] Table 3 Ablation experiments on TED and News datasets
[0113]
[0114] Table 3 shows that after removing the model corresponding to the method of the present invention, the s-BLUE and d-BLUE scores of the two data sets are significantly reduced, and the model performance has been greatly degraded, which proves the effectiveness and superiority of this method in paragraph-level neural machine translation.
[0115] The present invention proposes a globally consistent chapter-level neural machine translation method based on topic fusion. This method performs soft prompt processing on each topic word to effectively capture the global semantic information of the text and avoid topic drift. It also introduces a semantic consistency loss function guided by topic words to balance the contribution of source language information and topic information, avoid excessive reliance on topic words, and achieve a higher performance chapter-level neural machine translation system.
[0116] The specific implementation modes of the present invention are described in detail above in conjunction with the accompanying drawings, but the present invention is not limited to the above implementation modes, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.
Claims
1. A semantically consistent chapter-level neural machine translation method based on topic fusion, characterized in that: The following steps are involved: Step 1. Use the keyword extraction tool to extract key keywords from each chapter; Step 2: Load the results of sentence-level training to the chapter-level, and perform chapter-level machine translation training on the chapter-level data; Step 3: Introduce the topic migration mechanism and use Adapter to achieve the migration of sentence-level information to chapter-level information, and further optimize the chapter-level data; Step 4: Apply the semantic consistency loss function guided by keywords to optimize the model training process.
2. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 1, characterized in that: The step Step 1 includes: using a latent Dirichlet allocation (LDA) model to extract keywords from the chapter-level data.
3. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 2 is characterized by: The specific steps of using the latent Dirichlet allocation LDA model to extract topic words from the chapter-level data are as follows: Step 1.1, preprocess each chapter-level dataset to meet the input requirements of the LDA model, including removing stop words and low-frequency words; Step 1.2, input the preprocessed text data into the LDA model, set the number of topics T, and the model outputs the topic distribution of each chapter; Step 1.3: Select several high-probability words from the topic distribution of each chapter as subject words, and associate these subject words with the corresponding chapter data and save them for subsequent translation model training and topic transfer.
4. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 1, characterized in that: The specific steps of Step 2 are as follows: Step 2.1, load the trained sentence-level model parameters into the paragraph-level translation model to initialize the weights and configurations of the paragraph-level model; Step 2.2: Input the chapter-level data and extracted keywords into the chapter-level translation model with loaded parameters, and further optimize the model parameters through training, so that the model can improve the consistency and accuracy of translation based on the use of contextual information and topic information.
5. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 1, characterized in that: The step Step3 includes: adopting a prompt learning strategy guided by topic word soft prompts, in which each soft prompt is regarded as an "expert", and a hybrid expert model MoE is formed by the fusion of multiple soft prompts; in addition, by using the BERT model to encode the topic words, generating high-quality word vector representations, and introducing a topic word-aware multi-level dynamic fusion mechanism, these topic information are effectively integrated with the source language information, thereby achieving a topic migration effect.
6. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 1, characterized in that: The specific steps of Step 3 are as follows: Step 3.1, after obtaining the encoding representation H of the source language sentence m After obtaining the vector representation E of the topic word and the topic word, a prompt learning strategy guided by the topic word soft prompt is adopted to fuse the topic word information with the source language information; specifically, the information fusion is performed through the hybrid expert model MoE; first, the topic word vector representation is average pooled: Among them, E mean is the average representation of the subject words, T is the number of subject words, is the vector representation of the i-th topic word. Next, the MoE model is used to fuse the outputs of multiple "experts"; each "expert" corresponds to a specific topic representation, and the MoE model obtains the weight α of each expert through learning u , these expert outputs are fused into the final topic representation: Among them, α u It is the weight obtained through model learning, which represents the contribution of each expert to the final fusion result. In getting the keyword expression h MoE and the source language encoding H m Finally, the two types of information are further integrated through the gating mechanism; first, the gating value G is calculated: G=σ(W g ·[H m ,h MoE ]) Among them, σ represents the Sigmoid activation function, W g is the weight matrix of the fully connected layer, [H m ,h MoE ] represents the input vector obtained by performing element-by-element operations on the two; Finally, the gating value is used to represent the topic word h MoE and the source language encoding H m After fusion weighting, the final fusion expression is as follows: H fused =G⊙H m +(1-G)⊙h MoE Among them, ⊙ represents element-by-element multiplication; Step 3.2, introduce the Adapter module. The Adapter module is a lightweight bottleneck structure. By introducing the Adapter in the Transformer layer, it is used to capture and utilize the topic information. The process of the Adapter module is as follows: H adapted =Adapter(H fused ) Through the Adapter module, the model can better adapt to the topic information at different levels.
7. The semantic consistency chapter-level neural machine translation method based on topic fusion as claimed in claim 4, characterized in that: The step 4 includes the following: The consistency loss function is introduced, which aims to maintain the dominance of the source language information while effectively utilizing the information of the topic words. Specifically, the consistency loss is expressed as: Among them, λ is a weight coefficient used to balance the contribution of source language information and topic information.
Citation Information
Cited By
Large language model adaptive fusion method, device and equipment
CN120597991A
