A low-resource neural machine translation method fusing shared semantic space and bidirectional iterative generation optimization
By constructing a shared semantic space and a bidirectional iterative generation optimization module, the semantic alignment and overfitting problems in low-resource language translation are solved, achieving efficient improvement in cross-language translation quality.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- XI'AN POLYTECHNIC UNIVERSITY
- Filing Date
- 2026-03-31
- Publication Date
- 2026-06-30
AI Technical Summary
Existing neural machine translation models suffer from poor translation quality in low-resource language translation due to a lack of parallel corpora, and are prone to overfitting and semantic alignment problems.
A shared semantic space module and a bidirectional iterative generation and optimization module are constructed. The word vector space is aligned optimally through singular value decomposition and entropy regularization, and a bidirectional Transformer model is used for iterative generation and optimization to achieve cross-linguistic semantic consistency and model parameter convergence.
It significantly improves the accuracy, robustness, and consistency of long sentences in low-resource language translation, reduces training instability and overfitting, and enhances translation performance.
Smart Images
Figure CN122311241A_ABST
Abstract
Description
Technical Field
[0001] This invention belongs to the field of natural language processing and neural machine translation technology, and particularly relates to a low-resource neural machine translation method that integrates a shared semantic space and bidirectional iterative generation optimization. Background Technology
[0002] Neural machine translation (NMT), with its efficient semantic representation capabilities and excellent translation quality, has matured in high-resource language translation and has become an important tool for cross-language communication. However, for low-resource languages such as Tibetan, Mongolian, Korean, and Arabic, traditional translation models are prone to overfitting, semantic shift, and weak generalization due to the scarcity of parallel corpora, large lexical differences, and difficulties in semantic alignment. Currently, existing neural machine translation models, such as those based on the Transformer architecture, perform well in high-resource language translation tasks, but in low-resource language translation, the lack of parallel corpora makes it difficult to obtain effective training data, further resulting in poor translation quality.
[0003] In summary, current solutions primarily rely on data augmentation techniques, such as back-translation and data migration. However, these methods often fail to fully address the poor translation performance of low-resource languages, exhibiting issues such as overfitting. Therefore, a new approach is urgently needed to effectively mitigate semantic discrepancies during translation, improve the performance of low-resource language translation, and especially enhance its accuracy and robustness. Summary of the Invention
[0004] To address the problems existing in the prior art, this invention provides a low-resource neural machine translation method that integrates a shared semantic space and bidirectional iterative generation optimization.
[0005] This invention is implemented as follows: a low-resource neural machine translation method that integrates a shared semantic space and bidirectional iterative generation optimization, the method comprising:
[0006] The shared semantic space module and the bidirectional iterative generation optimization module (BIGO) are constructed as follows:
[0007] (1) Shared semantic space module:
[0008] Singular Value Decomposition (SVD) is used to decompose the product of the source language word vector matrix and the target language word vector matrix, yielding a left singular vector matrix U, a singular value diagonal matrix Σ, and a right singular vector matrix V. Based on the optimal solution to the Procrustes problem, an optimal orthogonal mapping matrix can be constructed from U and V. This is used to achieve initial alignment of the word vector spaces of the two languages;
[0009] Data preprocessing and shared vocabulary construction:
[0010] Cleaning, standardization, sub-word segmentation, and shared vocabulary construction are performed on Chinese and target low-resource languages to ensure that different language inputs have a consistent encoding format, which facilitates subsequent unified mapping to a shared semantic space.
[0011] Word vector space pre-alignment: The Procrustes method is used to minimize the spatial difference, and the alignment target is Equation (1):
[0012] (1)
[0013] Through Singular Value Decomposition (SVD): (2)
[0014] Then obtain That is, to find the linear transformation matrix that makes the word vector spaces of the two languages most consistent. ;
[0015] Entropy-regularized optimal transport (OT) cross-language alignment:
[0016] Using OT theory to measure the minimum transmission cost of semantic vector distribution between languages, the source vocabulary and extended vocabulary vectors are defined and expressed as formulas (3) to (6).
[0017] (3)
[0018] Source language word sequence and probability distribution:
[0019] (4)
[0020] Target language word sequence and probability distribution:
[0021] (5)
[0022] Formula (6): Semantic Alignment Space
[0023] (6)
[0024] The preference matrix and entropy regularization are described by formulas (7) to (9).
[0025] Formula (7): Calculate word vector similarity
[0026] (7)
[0027] Formulas (8) to (9): Information entropy is defined as...
[0028] (8)
[0029] (9)
[0030] The joint probability is independently determined by P(s) and P(e);
[0031] The overall objective function for optimal transmission optimization is expressed by formula (10):
[0032] (10)
[0033] Constructing a shared semantic representation:
[0034] The optimized mapping matrix is used in the word vector spaces of the two languages to express semantic similarity in the same semantic coordinate system, thereby achieving consistent cross-language representation; and providing low-resource languages with the ability to transfer knowledge from high-resource languages.
[0035] (2) Constructing the bidirectional iterative generative optimization model BIGO:
[0036] The bidirectional iterative generation optimization strategy is another core innovation of this invention. The core idea is to use a bidirectional Transformer model. In each iteration, the AdaGrad optimizer performs multiple rounds of forward generation, backward training, backward generation, and forward optimization during the back-translation process, simultaneously optimizing both the forward and backward Transformer models.
[0037] The weighted joint loss function of forward translation loss and backward reconstruction loss is shown in Equation (11):
[0038] (11)
[0039] in:
[0040] This is a weighted joint loss of the forward translation loss and the backward reconstruction loss, used to simultaneously update the parameters of the forward model and the backward model.
[0041] The translation loss from source language to target language;
[0042] Represents the target language → source language reconstruction loss;
[0043] Controlling the relative contributions of forward and reverse translation.
[0044] Based on the above technical solutions and the technical problems solved, the advantages and positive effects of the technical solutions protected by this invention are as follows:
[0045] This invention addresses key challenges in low-resource machine translation, such as corpus scarcity, semantic alignment difficulties, and model overfitting, by constructing a shared semantic space and introducing a bidirectional iterative generation optimization mechanism. The core approach involves first utilizing singular value decomposition and entropy regularization for optimal transport to map the source and target low-resource languages into a unified semantic space. This achieves high consistency between the two languages at the representation level, significantly improving the model's cross-language generalization ability under data-scarce conditions. Subsequently, through alternating training of forward and backward Transformer models, the invention continuously improves translation quality through a cycle of generating pseudo-bilingual corpora, reverse reconstruction, and joint forward and backward optimization. This allows the model to correct errors generated in each iteration and enhance semantic consistency. Ultimately, this invention achieves stable convergence of both model parameters and the semantic alignment matrix, significantly improving the accuracy, robustness, and long-sentence consistency of low-resource language translation, while effectively reducing training instability and overfitting. The overall solution improves translation performance without requiring a large amount of parallel corpora, offering significant advantages such as strong universality, high scalability, and applicability to various low-resource languages. Attached Figure Description
[0046] Figure 1 This is an overall architecture diagram of a translation system based on shared semantic space and bidirectional iterative generation optimization provided in an embodiment of the present invention;
[0047] Figure 2 This is a flowchart of the bidirectional iterative generation optimization (BIGO) mechanism provided in this embodiment of the invention;
[0048] Figure 3 This is a diagram of the forward and reverse Transformer collaborative training structure provided in an embodiment of the present invention. Detailed Implementation
[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.
[0050] Part 1: Construction of a Shared Semantic Space
[0051] Step 1: Corpus Preprocessing and Word Vector Training
[0052] First, parallel and monolingual corpora of Chinese and a low-resource language (such as Mongolian) are used, as follows:
[0053] Clean the text: remove symbols, spaces, and illegal encodings.
[0054] A unified sub-vocabulary (vocab_size=32k) is constructed using SentencePiece.
[0055] All text is segmented into sub-words, and language identifiers (such as...) are appended. <zh> 、 ).
[0056] Train 512-dimensional word vectors on word sequences of both languages respectively.
[0057] Step 2: Word Vector Space Pre-alignment
[0058] Parallel data is used to initialize cross-lingual word vectors, and semantic space alignment is achieved by optimizing the objective function. The Procrustes analysis method is employed.
[31] By minimizing the source language word vector matrix ( ) and the target language word vector matrix ( The Frobenius norm between ) is used to solve for the orthogonal projection matrix ( ), making as close as possible As shown in formula (1).
[0059] (1)
[0060] in and These are the word vector matrices for the source language and the target language, respectively. This represents the Frobenius norm, used to measure the difference between matrices. It is an orthogonal projection matrix used to align the source language word vector space to the target language word vector space.
[0061] This optimization problem has a closed-form solution, as shown in equations (2) and (3). This can be achieved by adjusting the matrix... Perform singular value decomposition (SVD).
[0062] (2)
[0063] (3)
[0064] in, These represent the standard components of singular value decomposition: the left singular vector matrix, the right singular vector matrix, and the singular value diagonal matrix. Formula (3) provides an optimal orthogonal transformation matrix that can effectively align source language word vectors to the target language word vector space, achieving minimum alignment error in the shared semantic space.
[0065] Step 3: Construct the OT alignment probability distribution
[0066] Word-level probability distribution definition: To construct the probability space for optimal transmission, token probabilities are defined for word sequences in the source and target languages respectively.
[0067] First, the vectors of the source vocabulary and the vectors of the expanded vocabulary:
[0068] (4)
[0069] The source language word sequence and probability distribution are shown in formula (3).
[0070] (5)
[0071] The target language word sequence and probability distribution are represented by formula (4).
[0072] (6)
[0073] Define the joint probability space:
[0074] (7)
[0075] The formula for calculating vector cosine similarity is as follows:
[0076] (8)
[0077] Information entropy:
[0078] (9)
[0079] The joint probability distribution satisfies:
[0080] (10)
[0081] Step 4: Entropy Regularization OT Optimization
[0082] The final shared semantic space will be constructed with the following objectives:
[0083] (11)
[0084] The optimal probability matrix is obtained by minimizing the optimal transmission loss. Used to update the semantic alignment matrix, so that the source language and the low-resource language achieve optimal semantic consistency in the same representation space.
[0085] Part Two: Bidirectional Iterative Generation Optimization Method
[0086] This invention aims to enable the model to more directly and fully integrate "forward" and "backward" contextual information. Therefore, the flowchart of the bidirectional iterative generative optimization model designed in this invention is shown in Figure 2. (Appendix) Figure 3 This is an architecture diagram combining back translation and Transformer. On top of the traditional Transformer module, this invention deploys a bidirectional translation network with a symmetrical structure in parallel: the forward network is responsible for processing the conversion from source language X to target language Y', while the backward network performs the reconstruction task from Y' to X'.
[0087] Details are as follows:
[0088] Forward model training:
[0089] A Transformer architecture with a 6-layer encoder and a 6-layer decoder is used. Parallel corpora are used to train the source language → target language translation to obtain initial translation capabilities. The loss is denoted as... .
[0090] Forward generation of pseudo-corpora:
[0091] Back-translate the source language monogram (input target language monogram corpus) to generate pseudo-translation (X, Y′).
[0092] Inverse model training:
[0093] The target language model is trained using pseudo-corpus, with a loss of [missing information]. .
[0094] Reverse generation of enhanced pseudo-corpora:
[0095] The inverse model is used to generate (Y, X′) by back-translating the target language monolingually, thus constructing an enhanced pseudo-corpus.
[0096] Iterative updates to the forward model:
[0097] The objective function is the weighted sum of the forward loss and the reverse loss, expressed as formula (12).
[0098] (12)
[0099] In the k-th iteration, the objective function's gradient is calculated using the AdaGrad optimizer, and the parameters of the three modules are updated: these are the parameters of the forward model. Inverse model parameters OT alignment matrix .
[0100] Multiple iterations until convergence:
[0101] After multiple rounds of iterative optimization, the parameters of the forward model converged, meaning that BLEU and Loss_fwd no longer showed significant improvement. The shared semantic space stabilized, indicating the optimal transport distribution. No longer changing significantly, alignment matrix convergence.
[0102] Part Three: Model Training, Experiment Setup, and Results
[0103] The experimental datasets are based on the IWSLT and CCMT2018 datasets, covering multiple low-resource language pairs such as Chinese–Tibetan, Chinese–Mongolian, and Chinese–Arabic.
[0104] The model configuration adopts a Transformer model structure with 6 layers of Encoder and 6 layers of Decoder; word embedding dimension is 512, attention head number is 8; Adam optimizer is used, and the learning rate is set to 1e−3; the batch size is 64 samples per batch.
[0105] Experimental results: In the Chinese-Tibetan language task, the method of this invention improves the BLEU score by about 15% and the F1 score by 10%; in the Chinese-Arabic language task, the BLEU score improves by about 12%, and the model shows significant improvement in the COMET metric; compared with the traditional Transformer and mBART models, the proposed solution shows higher stability and generalization ability in low-resource environments.
[0106] In one exemplary implementation, a forward translation model from the source language to the target language and a backward translation model from the target language to the source language are constructed. These two models can respectively employ an encoder-decoder network based on a Transformer structure, a lightweight attention network, or a sequence-to-sequence network incorporating a gating mechanism. During parameter initialization, the forward and backward models can employ independent initialization, partially shared initialization, or fully shared embedding layers to adapt to training stability requirements under different low-resource corpus sizes.
[0107] In the bidirectional iterative optimization process, each iteration includes: first, generating a target language translation from the source language sentence using the current forward translation model; then, using this translation as pseudo-parallel data to update the parameters of the backward translation model once or multiple times; finally, generating the source language reconstruction result from the updated backward model, and using this result to perform backward training and updates on the forward translation model. The above cyclical update process can be set to a fixed number of iterations (e.g., 5, 10, or 20 rounds), or it can be adaptively terminated based on the convergence of the joint loss function.
[0108] The joint loss function is formed by a weighted combination of forward translation loss and backward reconstruction loss. The forward translation loss can be cross-entropy loss, labeled smoothing cross-entropy loss, or translation loss with coverage constraints. The backward reconstruction loss can be sequence reconstruction error or an approximate loss based on edit distance. The weight parameters of the two types of losses can be set to different value ranges such as 0.3, 0.5, and 0.7, and are dynamically adjusted during training to balance translation fluency and semantic consistency.
[0109] During the model parameter update phase, an adaptive gradient descent optimization strategy, such as Adam, AdamW, or RMSProp with momentum, is preferred. This is combined with learning rate warm-up, gradient pruning, and parameter regularization strategies to improve the convergence stability of the bidirectional model under low resource conditions.
[0110] In another exemplary embodiment, the shared semantic space construction mechanism is synergistically integrated with the aforementioned bidirectional iterative generation optimization mechanism. Specifically, the source and target language inputs are first mapped to a unified semantic space through a shared word embedding space, aligned sub-word encoding, or a cross-lingual contrastive learning network, and then input into the bidirectional translation model for training. High-confidence translations generated during the bidirectional translation process can be filtered through an output probability threshold or an uncertainty assessment strategy, and fed back to update the mapping parameters of the shared semantic space.
[0111] By alternately updating the shared semantic space and bidirectional translation model after each iteration, a synergistic enhancement of cross-linguistic semantic alignment and translation generation capabilities is achieved. This synergistic mechanism is not a simple patchwork of existing technologies, but rather forms a stable closed-loop optimization path under the joint constraints of bidirectional generation, semantic alignment, and loss. This significantly improves translation accuracy, robustness, and generalization ability in low-resource scenarios, demonstrating outstanding substantive features and significant technological progress.
[0112] Example 1: Cross-language construction and stable alignment of shared semantic spaces
[0113] Chinese is selected as the source language, and a low-resource language is selected as the target language. Independent word vector models are trained separately to obtain word vector matrices for the source language and the target language. Matrix multiplication is performed on the two matrices to form a cross-language correlation matrix, and singular value decomposition is performed on it to obtain left and right singular vector matrices. Based on these matrices, an orthogonal mapping matrix is constructed to map the word vectors of the source and target languages to a unified semantic coordinate system, thereby achieving preliminary semantic alignment. This method avoids dependence on manual dictionaries, achieves fully automatic structured alignment, and demonstrates the method's scalability and low-resource adaptability.
[0114] Example 2: Semantic Alignment Optimization Based on Probability Distribution and Similarity
[0115] The frequency of word occurrences in both the source and target language word sets is statistically analyzed and normalized to form a probability distribution. The inner product similarity of cross-lingual word vectors in a unified coordinate system is calculated and normalized to construct a preference matrix. Joint optimization is performed with the goal of minimizing the overall semantic transmission cost and maximizing the joint distribution entropy, thereby obtaining the cross-lingual semantic mapping relationship. This process ensures that alignment considers not only geometric proximity but also introduces distribution consistency constraints, significantly improving the stability of semantic matching and demonstrating the advancement of this method compared to simple linear alignment.
[0116] Example 3: Closed-loop training process generated by bidirectional iteration
[0117] A forward translation model and a backward translation model are constructed. In each training round, the forward model first generates the target sentence, and then uses this sentence to train the backward model; then the backward model generates the source sentence, which is used to update the forward model. The forward translation loss and the backward reconstruction loss are combined with weights to form a joint loss function, which updates the parameters of both models simultaneously. This bidirectional closed loop enables the model to learn not only the translation mapping but also the reversible semantic structure, effectively reducing the risk of overfitting under low-resource conditions.
[0118] Example 4: Collaborative Feedback Mechanism of Shared Semantic Space and Bidirectional Generation
[0119] In the bidirectional generation process, high-confidence translation pairs are selected and used as feedback to update the semantic mapping relationships in the shared semantic space, causing the shared space to converge toward the true semantic distribution round by round. The updated shared space then serves as the input basis for the next round of bidirectional generation, thus forming a positive feedback loop between alignment and generation. This collaborative mechanism breaks through the traditional unidirectional process of "align first, then train," achieving the joint evolution of the semantic space and the translation model.
[0120] Example 5: Performance Improvement Verification in Low-Resource Scenarios
[0121] Experiments were conducted using only a small-scale parallel corpus, comparing the effects of using only a shared semantic space, using only bidirectional generation, and using both simultaneously. The results show that the scheme using both shared semantic space and bidirectional generation outperforms the single scheme in terms of translation accuracy, stability, and convergence speed, indicating a non-linear gain relationship between the two mechanisms and demonstrating their synergistic innovation.
[0122] Example 6: Verification of Cross-Language Extension and Generalization Capabilities
[0123] This method is extended from bilingual to multilingual scenarios, maintaining a shared semantic space as a unified hub. This allows multiple low-resource languages to be simultaneously mapped to the same high-resource semantic space, where bidirectional generative training is performed. Experimental results show that new languages require only a small amount of corpus to be integrated into the system and acquire usable translation capabilities, validating the engineering value of this approach in multilingual transfer, knowledge reuse, and system expansion.
[0124] Through a collaborative design of shared semantic space and bidirectional iterative generation, this method achieves a triple fusion of structural alignment, probabilistic alignment, and generative optimization. This enables the system to align not only the geometric structure of word vectors but also their semantic distribution, and to continuously self-correct during the generation process. This overall approach significantly improves translation performance under low-resource conditions, while possessing the combined advantages of automation, scalability, and stable convergence, demonstrating remarkable and substantial progress.
[0125] It should be noted that embodiments of the present invention can be implemented in hardware, software, or a combination of both. The hardware portion can be implemented using dedicated logic; the software portion can be stored in memory and executed by a suitable instruction execution system, such as a microprocessor or dedicated-design hardware. Those skilled in the art will understand that the above-described devices and methods can be implemented using computer-executable instructions and / or included in processor control code, for example, such code provided on a carrier medium such as a disk, CD, or DVD-ROM, a programmable memory such as read-only memory (firmware), or a data carrier such as an optical or electronic signal carrier. The devices and modules of the present invention can be implemented by hardware circuitry such as very large-scale integrated circuits or gate arrays, semiconductors such as logic chips, transistors, or programmable hardware devices such as field-programmable gate arrays, programmable logic devices, etc., or by software executed by various types of processors, or by a combination of the above-described hardware circuitry and software, such as firmware.
[0126] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any modifications, equivalent substitutions, and improvements made by those skilled in the art within the scope of the technology disclosed in the present invention, and within the spirit and principles of the present invention, should be covered within the scope of protection of the present invention. < / zh>
Claims
1. A low-resource neural machine translation method that integrates shared semantic space and bidirectional iterative generation optimization, characterized in that, This includes a shared semantic space construction mechanism, which comprises the following steps: Perform matrix multiplication on the source language word vector matrix and the target language word vector matrix to form a cross-language correlation matrix; The cross-language correlation matrix is subjected to singular value decomposition to obtain a left singular vector matrix and a right singular vector matrix; An orthogonal mapping matrix is constructed based on the left singular vector matrix and the right singular vector matrix to align the word vector space of the source language with the word vector space of the target language in a unified coordinate system. Construct probability distributions based on word occurrence frequency normalization for both the source language word set and the target language word set; Construct a preference matrix consisting of the similarity between word vectors in the source language and word vectors in the target language; Based on the probability distribution and preference matrix, an entropy-regularized optimal transmission objective function is constructed to minimize the transmission cost of the semantic distribution and maximize the distribution entropy. By solving the entropy-regularized optimal transmission objective function, cross-language semantic mapping relationships are obtained, and a shared semantic representation space is constructed.
2. The method according to claim 1, characterized in that, The word vector similarity is obtained by calculating the vector dot product and normalizing the vector magnitude.
3. The method according to claim 1, characterized in that, The probability distribution is composed of the proportion of each word in the corpus, such that the sum of the probabilities of all words is 1.
4. The method according to claim 1, characterized in that, The entropy regularization optimal transmission objective function simultaneously constrains the minimization of total transmission cost and the maximization of joint distribution uncertainty.
5. A bidirectional iterative method for generating optimized neural machine translation, characterized in that, This includes a bidirectional iterative generation optimization mechanism, which includes: Construct a forward translation model from the source language to the target language and a reverse translation model from the target language to the source language; In each iteration, a cyclical update process of forward translation generation, reverse model training, and reverse translation generation and forward model training is performed; Construct a joint loss function formed by a weighted combination of forward translation loss and backward reconstruction loss; The parameters of both the forward and backward translation models are updated simultaneously based on the joint loss function, thereby achieving collaborative convergence of the bidirectional models.
6. The method according to claim 5, characterized in that, The weights of the forward translation loss and the backward reconstruction loss in the joint loss function are adjustable.
7. The method according to claim 5, characterized in that, The model parameter updates employ an adaptive gradient descent optimization strategy.
8. A low-resource neural machine translation collaborative method that integrates shared semantic space and bidirectional iterative generation optimization, characterized in that, This includes the following collaborative mechanisms: The shared semantic space construction mechanism described in claim 1 is used to perform unified semantic representation of source language and target language inputs; The unified semantic representation is input into the bidirectional iterative generation optimization mechanism described in claim 5 for bidirectional translation training; The high-confidence translations generated by bidirectional translation training are fed back into the shared semantic space construction mechanism to update the semantic mapping relationship; By alternating between shared semantic space updates and bidirectional translation model updates, joint optimization of cross-linguistic semantic alignment and translation generation is achieved.
9. The method according to claim 8, characterized in that, The high-confidence translations are obtained by filtering the model output probability threshold.
10. The method according to claim 8, characterized in that, The joint optimization process updates the shared semantic space and the bidirectional translation model after each iteration.