Parallel sentence pair extraction method and device based on romanization feature enhancement large model

Abstract

This invention relates to a method and apparatus for extracting parallel sentence pairs from a large model based on Romanization feature enhancement, belonging to the field of natural language processing. The invention includes: firstly, data preprocessing to obtain a training instruction set, a bilingual translation instruction set, and a parallel sentence pair extraction instruction set; training a large model for parallel sentence pair extraction based on Romanization feature enhancement: firstly, incorporating Romanized language features into the large language model; then, enhancing the alignment ability between Chinese and Thai / Lao through translation task training; secondly, performing supervised fine-tuning to optimize the semantic mapping ability of the target language pair in the large language model through explicit alignment signals; finally, using the parallel sentence pair extraction instruction set and employing a multi-task joint training strategy for LoRA incremental fine-tuning; and finally, constructing a large model for parallel sentence pair extraction based on Romanization feature enhancement. This invention significantly improves the performance of Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks.

Description

Technical Field

[0001] This invention relates to a method and apparatus for extracting large-scale parallel sentence pairs based on Romanization feature enhancement, belonging to the field of natural language processing technology. Background Technology

[0002] Training for multilingual large-scale model translation tasks relies on parallel corpora from various translation directions. The quantity and quality of these corpora directly determine the performance of multilingual neural machine translation. Due to the rich parallel corpora of high-resource language pairs such as English-German and English-French, machine translation has demonstrated extremely high performance on these language pairs, almost approaching human translation levels. However, the machine translation performance of low-resource languages ​​such as Thai and Lao is severely limited due to the scarcity of parallel corpora. Research shows that online resources from different application domains can effectively expand the corpora of low-resource languages. Thai and Lao mainly exist on the internet in the form of comparable corpora; extracting high-quality parallel data from these corpora is one effective way to alleviate the scarcity of Chinese-Thai and Chinese-Lao parallel corpora. Methods for extracting low-resource parallel sentence pairs using comparable corpora have achieved good results, demonstrating that this parallel data can significantly improve machine translation performance.

[0003] The key to parallel sentence pair extraction is matching two languages ​​based on semantic similarity, with the core being the alignment of bilingual semantic spaces. The goal of this task is to leverage the extracted parallel sentence pairs to provide data support for other NLP tasks. Traditional semantic space alignment methods typically rely on different neural network architectures to learn sentence vector representations for each language and map them to a shared vector space to evaluate the semantic similarity of cross-lingual sentence pairs. However, traditional semantic space alignment methods are limited by the quality and coverage of training data, making it difficult to generate accurate semantic representations and affecting the semantic alignment results. Numerous studies have shown that large language models based on decoder-only architectures, trained on large-scale multilingual corpora, can capture commonalities between languages, generate consistent cross-lingual semantic representations, extract rich semantic information, and cover complex semantic relationships at the sentence level. Compared to traditional neural network models, these large language models, with their large number of parameters, can better model the diversity and complexity of languages, producing more accurate and cross-lingual adaptive semantic representations. Furthermore, the flexible input paradigm of large language models supports dynamic adjustment of the inference process, significantly improving the performance of semantic alignment tasks. Summary of the Invention

[0004] The technical problem to be solved by this invention is that it provides a large-scale parallel sentence pair extraction method based on Romanization feature enhancement to address the problem of insufficient representation ability of sentence pair extraction models due to the scarcity of Thai and Lao bilingual alignment data. This invention can construct high-quality Thai-Chinese and Lao-Chinese parallel data.

[0005] The technical solution of the present invention is as follows: Firstly, the present invention provides a method for extracting parallel sentence pairs from a large model based on Romanization feature enhancement, the method comprising:

[0006] Step 1: Data Preprocessing

[0007] The texts of Thai and Lao were converted into Romanized characters using a Romanization conversion tool, and a Thai-Romanized and Lao-Romanized training instruction set was constructed.

[0008] Based on open-source corpora and comparable Internet corpora, we constructed English-Thai and English-Lao bilingual translation instruction sets and Chinese-Thai and Chinese-Lao parallel sentence pair extraction instruction sets;

[0009] Step 2: Training a large-scale parallel sentence pair extraction model based on Romanization feature enhancement: First, Romanization language features are integrated into the large language model;

[0010] Then, the alignment ability between Chinese and Thai / Lao is enhanced through translation task training;

[0011] Secondly, supervised fine-tuning is performed, using parallel sentence pairs to extract instruction sets and optimize the semantic mapping capability of the target language pairs of the large language model through explicit alignment signals;

[0012] Finally, the training instruction set constructed in Step 1 is used to perform LoRA incremental fine-tuning using a multi-task joint training strategy;

[0013] Step 3: Construct a large-scale parallel sentence pair extraction device based on Romanization feature enhancement: First, deploy the large-scale parallel sentence pair extraction model based on Romanization feature enhancement trained in Step 2 to the server and provide the corresponding service interface; then, construct a visualization platform for Chinese-Lao-Tai parallel sentence pair extraction; finally, obtain the corresponding parallel sentence pair extraction results by calling the model interface deployed on the server, thus forming a large-scale parallel sentence pair extraction device based on Romanization feature enhancement for parallel sentence pair extraction.

[0014] Further, Step 1 includes:

[0015] Step 1.1, Data Construction: First, based on the Romanization conversion tool, the open-source corpus OPUS, and comparable corpora collected from the Internet, construct the instruction set required for multi-task joint training;

[0016] Specifically, this includes: a training instruction set for Thai-Romanization and Lao-Romanization; a bilingual translation instruction set for English-Thai and English-Lao to support training for English-Thai and English-Lao translation tasks; and a parallel sentence pair extraction instruction set for Chinese-Thai and Chinese-Lao.

[0017] Step 1.2, Data Processing: After the initial construction is completed, all corpora are processed uniformly, including deduplication to avoid duplicate samples interfering with the training effect; at the same time, samples containing missing values ​​are removed to ensure the integrity and consistency of the corpus.

[0018] Step 1.3, Instruction Set Construction: Finally, the processed corpus is divided into training set, validation set and test set to form the instruction set required for multi-task joint training.

[0019] Furthermore, Step 2 includes:

[0020] Step 2.1: Train the large language model to fully learn the knowledge of Tai Lao Romanization and enhance cross-language relevance by utilizing its shared character space;

[0021] Step 2.2: By leveraging a large-scale parallel corpus of English, the large language model implicitly learns the syntactic-semantic mapping rules between English and Thai / Lao through joint training of "English → Thai" and "English → Lao" translation tasks, thereby enhancing the alignment ability between Chinese and Thai / Lao.

[0022] Step 2.3: Use Chinese-Thai and Chinese-Lao parallel sentences to perform supervised fine-tuning of the extracted instruction set, and optimize the target language semantic mapping capability of the model by explicitly aligning signals;

[0023] Step 2.4: Use a multi-task joint training strategy to perform incremental fine-tuning of LoRA.

[0024] Furthermore, the specific processing flow of Step 2 includes:

[0025] The input to a large language model that only decodes is a sequence. , m represents the length of the tokens in the input sequence, and each token Through word embedding matrix Convert to vector , with position encoding Overlay:

[0026]

[0027] in, Represents token The corresponding embedding vector; Indicates the vocabulary size of the model Indicates the length of the vector converted from each token; position encoding The calculation formula is:

[0028]

[0029]

[0030] in, The positional encoding vector takes values ​​in even-numbered dimensions, represented by a sine function; The positional encoding vector takes values ​​in odd-numbered dimensions, represented by a cosine function; This represents the dimension index in the positional encoding vector;

[0031] Obtain the initial embedding matrix The matrix is ​​input to the system by In an architecture where Transformer blocks are stacked, multi-head self-attention calculation of the mask is performed. Projected to Subspace:

[0032]

[0033] in, express The output of the layer Transformer block, , , Let represent the query, key, and value of the j-th projection subspace, respectively. , , Let represent the weights of the query, key, and value in the j-th projection subspace, respectively. , Indicates the number of attention heads in multi-head attention; the output of each single-head attention. for:

[0034]

[0035] in, Indicates the first A person's attention Matrix weights This indicates that the output of the attention head is Softmax normalized; mask matrix It is to ensure the position Accessible only The token is obtained by concatenating multiple outputs and then performing a linear transformation:

[0036]

[0037] in, This represents the final output of the attention layer. This indicates that the outputs of multi-head attention are spliced ​​together. Indicates the output projection matrix; performs residual connections and layer normalization:

[0038]

[0039] in, This represents the representation after performing the multi-head attention mechanism, followed by residual connections and layer normalization. This indicates that a layer normalization operation is being performed;

[0040] The feedforward network (FNN) performs a nonlinear transformation on the normalized result:

[0041]

[0042] in, This represents the output after passing through the FFN layer; It is a commonly used nonlinear activation function; , This represents the two weight matrices of a feedforward network (FFN). , Represent the two bias vectors of the feedforward network FFN;

[0043] The residuals are then reconnected and normalized to obtain the output of the current layer. :

[0044]

[0045] go through After layers are stacked, the final hidden state By outputting the projection matrix Convert to word probability distribution :

[0046]

[0047] Each location Word probability distribution for:

[0048]

[0049] in, Indicates a given Under the condition of model prediction The probability of;

[0050] The model training uses weighted maximum likelihood estimation as the model's loss function. :

[0051]

[0052] in, Represents the input sequence; Represents the target sequence; Indicates the distribution of the training data; Represents the parameters of the model; This indicates that the model performs well under given input. and parameters Predict The conditional probability; Represents the input sequence The weighting function is used to adjust the importance of different samples; Represents the input sequence and target sequence The maximum likelihood.

[0053] Furthermore, the specific steps of Step 3 include:

[0054] Step 3.1: Deploy the large model using the Xinference open-source framework to the server. The trained parallel sentence pair extraction model based on Romanization feature enhancement is deployed to the server. Xinference provides a unified API interface to call the model.

[0055] Step 3.2: Use Python's Streamlit framework to build a visualization platform for extracting parallel sentence pairs in Chinese, Lao, and Thai languages. Users input sentences to be compared in both languages ​​through the visualization interface, and send the input text information to the server through the API interface. Finally, the parallel sentence pair extraction model based on the Romanization feature enhancement performs inference to obtain the parallel sentence pair extraction results and returns them to the user.

[0056] Secondly, the present invention provides an apparatus for extracting large model parallel sentence pairs based on Romanization feature enhancement, the apparatus comprising: a module for performing the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0057] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0058] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0059] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0060] The beneficial effects of this invention are:

[0061] 1. This invention first introduces Romanized language features into a multilingual large model to enhance the semantic representation of Thai and Lao. Secondly, it uses a customized English-centered translation instruction set strategy to bring the semantic distance between Chinese and Lao / Thai languages ​​closer, thereby improving the model's cross-linguistic representation ability. Finally, it uses Chinese-Thai / Lao parallel sentences to incrementally fine-tune the extraction instruction set, further optimizing the extraction accuracy of Thai-Chinese and Lao-Chinese parallel sentence pairs.

[0062] 2. Experiments show that the method of the present invention significantly improves the performance of Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks, and the present invention can construct high-quality Thai-Chinese and Lao-Chinese parallel data. Attached Figure Description

[0063] Figure 1 A Chinese-Lao-Thai parallel sentence pair extraction architecture driven by large-model-based Romanization cross-linguistic feature fusion and strategy enhancement.

[0064] Figure 2 This is a flowchart of the method in this invention. Detailed Implementation

[0065] Example 1: As Figures 1-2 As shown, in a first aspect, the present invention provides a method for extracting parallel sentence pairs from large models based on Romanization feature enhancement, the method comprising:

[0066] Step 1: Data Preprocessing

[0067] The texts of Thai and Lao were converted into Romanized characters using a Romanization conversion tool, and a Thai-Romanized and Lao-Romanized training instruction set was constructed.

[0068] Based on open-source corpora and comparable Internet corpora, we constructed English-Thai and English-Lao bilingual translation instruction sets and Chinese-Thai and Chinese-Lao parallel sentence pair extraction instruction sets;

[0069] Further, Step 1 includes:

[0070] Step 1.1, Data Construction: First, based on the Romanization conversion tool, the open-source corpus OPUS, and comparable corpora collected from the Internet, construct the instruction set required for multi-task joint training;

[0071] Specifically, this includes: a training instruction set for Thai-Romanization and Lao-Romanization; a bilingual translation instruction set for English-Thai and English-Lao to support training for English-Thai and English-Lao translation tasks; and a parallel sentence pair extraction instruction set for Chinese-Thai and Chinese-Lao.

[0072] Step 1.2, Data Processing: After the initial construction is completed, all corpora are processed uniformly, including deduplication to avoid duplicate samples interfering with the training effect; at the same time, samples containing missing values ​​are removed to ensure the integrity and consistency of the corpus.

[0073] Step 1.3, Instruction Set Construction: Finally, the processed corpus is divided into training set, validation set and test set to form the instruction set required for multi-task joint training.

[0074] Step 2: Training a large-scale parallel sentence pair extraction model based on Romanization feature enhancement: First, Romanization language features are integrated into the large language model;

[0075] Then, the alignment ability between Chinese and Thai / Lao is enhanced through translation task training;

[0076] Secondly, supervised fine-tuning is performed, using parallel sentence pairs to extract instruction sets and optimize the semantic mapping capability of the target language pairs of the large language model through explicit alignment signals;

[0077] Finally, the training instruction set constructed in Step 1 is used to perform LoRA incremental fine-tuning using a multi-task joint training strategy;

[0078] Furthermore, Step 2 includes:

[0079] Step 2.1: Train the large language model to fully learn the knowledge of Tai Lao Romanization and enhance cross-language relevance by utilizing its shared character space;

[0080] Step 2.2: By leveraging a large-scale parallel corpus of English, the large language model implicitly learns the syntactic-semantic mapping rules between English and Thai / Lao through joint training of "English → Thai" and "English → Lao" translation tasks, thereby enhancing the alignment ability between Chinese and Thai / Lao.

[0081] Step 2.3: Use Chinese-Thai and Chinese-Lao parallel sentences to perform supervised fine-tuning of the extracted instruction set, and optimize the target language semantic mapping capability of the model by explicitly aligning signals;

[0082] Step 2.4: Use a multi-task joint training strategy to perform incremental fine-tuning of LoRA.

[0083] Furthermore, the specific processing flow of Step 2 includes:

[0084] The input to a large language model that only decodes is a sequence. , m represents the length of the tokens in the input sequence, and each token Through word embedding matrix Convert to vector , with position encoding Overlay:

[0085]

[0086] in, Represents token The corresponding embedding vector; Indicates the vocabulary size of the model Indicates the length of the vector converted from each token; position encoding The calculation formula is:

[0087]

[0088]

[0089] in, The positional encoding vector takes values ​​in even-numbered dimensions, represented by a sine function; The positional encoding vector takes values ​​in odd-numbered dimensions, represented by a cosine function; This represents the dimension index in the positional encoding vector;

[0090] Obtain the initial embedding matrix The matrix is ​​input to the system by In an architecture where Transformer blocks are stacked, multi-head self-attention calculation of the mask is performed. Projected to Subspace:

[0091]

[0092] in, express The output of the layer Transformer block, , , Let represent the query, key, and value of the j-th projection subspace, respectively. , , Let represent the weights of the query, key, and value in the j-th projection subspace, respectively. , Indicates the number of attention heads in multi-head attention; the output of each single-head attention. for:

[0093]

[0094] in, Indicates the first A person's attention Matrix weights This indicates that the output of the attention head is Softmax normalized; mask matrix It is to ensure the position Accessible only The token is obtained by concatenating multiple outputs and then performing a linear transformation:

[0095]

[0096] in, This represents the final output of the attention layer. This indicates that the outputs of multi-head attention are spliced ​​together. Indicates the output projection matrix; performs residual connections and layer normalization:

[0097]

[0098] in, This represents the representation after performing the multi-head attention mechanism, followed by residual connections and layer normalization. This indicates that a layer normalization operation is being performed;

[0099] The feedforward network (FNN) performs a nonlinear transformation on the normalized result:

[0100]

[0101] in, This represents the output after passing through the FFN layer; It is a commonly used nonlinear activation function; , This represents the two weight matrices of a feedforward network (FFN). , Represent the two bias vectors of the feedforward network FFN;

[0102] The residuals are then reconnected and normalized to obtain the output of the current layer. :

[0103]

[0104] go through After layers are stacked, the final hidden state By outputting the projection matrix Convert to word probability distribution :

[0105]

[0106] Each location Word probability distribution for:

[0107]

[0108] in, Indicates a given Under the condition of model prediction The probability of;

[0109] The model training uses weighted maximum likelihood estimation as the model's loss function. :

[0110]

[0111] in, Represents the input sequence; Represents the target sequence; Indicates the distribution of the training data; Represents the parameters of the model; This indicates that the model performs well under given input. and parameters Predict The conditional probability; Represents the input sequence The weighting function is used to adjust the importance of different samples; Represents the input sequence and target sequence The maximum likelihood.

[0112] Step 3: Construct a large-scale parallel sentence pair extraction device based on Romanization feature enhancement: First, deploy the large-scale parallel sentence pair extraction model based on Romanization feature enhancement trained in Step 2 to the server and provide the corresponding service interface; then, construct a visualization platform for Chinese-Lao-Tai parallel sentence pair extraction; finally, obtain the corresponding parallel sentence pair extraction results by calling the model interface deployed on the server, thus forming a large-scale parallel sentence pair extraction device based on Romanization feature enhancement for parallel sentence pair extraction.

[0113] Furthermore, the specific steps of Step 3 include:

[0114] Step 3.1: Deploy the large model using the Xinference open-source framework to the server. The trained parallel sentence pair extraction model based on Romanization feature enhancement is deployed to the server. Xinference provides a unified API interface to call the model.

[0115] Step 3.2: Use Python's Streamlit framework to build a visualization platform for extracting parallel sentence pairs in Chinese, Lao, and Thai languages. Users input sentences to be compared in both languages ​​through the visualization interface, and send the input text information to the server through the API interface. Finally, the parallel sentence pair extraction model based on the Romanization feature enhancement performs inference to obtain the parallel sentence pair extraction results and returns them to the user.

[0116] Secondly, the present invention provides an apparatus for extracting large model parallel sentence pairs based on Romanization feature enhancement, the apparatus comprising: a module for performing the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0117] Thirdly, the present invention provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0118] Fourthly, the present invention provides a non-transitory computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0119] Fifthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the large model parallel sentence pair extraction method based on Romanization feature enhancement.

[0120] To verify the effectiveness of the proposed invention, experiments were conducted in a Linux system environment. The LLaMA Factory framework (version 0.10.2, Python 3.9) was used, and the LoRA method was employed for supervised fine-tuning (SFT) of the model. The rank was set to 8, and the target module was set to "all," meaning LoRA was applied to all linear layers. The training hardware configuration consisted of four 3090 GPUs, with a training batch size of 4 and gradient accumulation steps of 16 per GPU; the validation batch size was 2 per GPU. The maximum length of the input sequence was set to `cutoff_len=512`. The optimizer used was AdamW, with an initial learning rate of 5e-5. The learning rate scheduler employed a cosine annealing strategy, and the warmup ratio was set to 0.1 to stabilize the learning process in the early stages of training. Weight decay and random deactivation were set to their default values ​​of 0.0 and 0.1, respectively. To prevent overfitting, an early stopping mechanism was enabled, automatically saving the best model when the validation set performance no longer improved. The total number of training epochs was 5.0, and the validation set was split from the training set at a ratio of 10%. During training, logs are recorded every 100 steps, and the model is validated and saved every 600 steps. The `plot_loss` function is enabled to draw loss curves for easy monitoring of training dynamics. To improve training efficiency and reduce GPU memory usage, FP16 mixed-precision training is enabled.

[0121] Experiment 1: Comparative Experiment;

[0122] To verify the effectiveness of the proposed method, this paper compares the proposed method with existing baseline models, and the specific experimental results are shown in Table 1.

[0123] (1) SVM: As a classic algorithm in supervised learning frameworks, support vector machines achieve binary classification by constructing the optimal hyperplane. This invention applies it to the discrimination task of Chinese-Thai and Chinese-Lao bilingual parallel corpora, and constructs classification models in the two language pairs respectively.

[0124] (2) LR: As a statistical learning method based on probability discrimination, logistic regression maps linear combinations to classification probabilities through the sigmoid function. This invention constructs regression models on Chinese-Thai and Chinese-Lao bilingual text pairs respectively.

[0125] Verify its linear classification ability under high-dimensional sparse features.

[0126] (3) Bi-RNN: Bidirectional Long Short-Term Memory Network, which captures contextual dependencies through forward and backward hidden state fusion. It enables semantic classification of Chinese-Thai and Chinese-Lao bilingual texts.

[0127] (4) mBERT: Use a multilingual BERT pre-trained model to obtain semantic representations, calculate the cosine similarity between Chinese-Thai / Chinese-Lao sentence vectors, and realize the classification of Chinese-Thai and Chinese-Lao parallel sentence pairs.

[0128] (5) Bi-LSTM+mBERT: Based on the hierarchical feature extraction strategy, cross-language representations are obtained through a pre-trained encoder, and then long-distance semantic associations are captured through a bidirectional gated recurrent network. Finally, multi-granular features are fused to complete the parallelism determination.

[0129] (6) LaBSE: A dual-tower architecture based on multilingual contrastive learning, which achieves cross-lingual vector space alignment through supervised training via translation. The LaBSE model obtains language-independent sentence representations and determines whether sentence pairs are parallel.

[0130] (7) E5: Based on the text pair enhancement and contrast loss optimization strategy, discriminative embedding representations are learned through the InfoNCE objective function. Parallel sentence pairs are classified through semantic space projection.

[0131] (8) Literature method: Data augmentation was performed using a Thai-Lao bilingual dictionary. Contrastive learning was designed in the Siamese network and a joint training strategy was introduced to enhance the consistency of model representation and achieve parallel sentence pair extraction.

[0132] (9) hfl-lllama3-it: Use the Chinese-centric hfl:llama-3-chinese-8b-instruct model for zero-shot parallel sentence pair extraction task.

[0133] (10) hfl-llama3-base-sft: The method of the present invention is used to train on the Base model of hfl-llama3 and to extract parallel sentence pairs.

[0134] (11) hfl-llama3-it-sft: The method of the present invention is used to train on the instruction version of hfl-llama3 and to extract parallel sentence pairs.

[0135] Table 1 compares the performance of parallel sentence extraction tasks in Hantai and Hanlao.

[0136]

[0137] Experimental results show that the proposed method for extracting parallel sentence pairs between Chinese, Lao, and Thai based on Romanization feature enhancement significantly outperforms traditional machine learning, neural networks, and existing baseline models on both Chinese-Thai and Chinese-Lao tasks. As shown in Table 1, traditional machine learning methods (SVM, LR) achieve F1 scores below 70% for both language pairs, with the F1 score for the Chinese-Lao task being only 54.73%-55.80%, reflecting the insufficient representation ability of traditional models for low-resource languages. Bi-RNN, by modeling sequence features through a bidirectional recurrent neural network, improves the F1 scores for the Chinese-Thai and Chinese-Lao tasks to 76.99% and 61.36%, respectively. However, limited by its local semantic capture capability, it still cannot effectively address the cross-linguistic semantic gap problem. The pre-trained models mBERT and Bi-LSTM+mBERT achieved F1 scores of 76.45%-83.90% on the Chinese-Tai task, but only 55.38%-66.57% on the Chinese-Old task, indicating that relying solely on multilingual pre-trained models faces cross-language transfer bottlenecks in scenarios with scarce bilingual data. The LaBSE and E5 models based on contrastive learning significantly improved performance through language-independent semantic representations. LaBSE achieved an F1 score of 93.18% on the Chinese-Old task, but its F1 score on the Chinese-Tai task (90.19%) was lower than the methods in the literature (91.80%). This suggests that relying solely on general semantic similarity calculations is insufficient to fully model the special relationships between low-resource languages. The literature further optimized the model through bilingual dictionary enhancement and contrastive learning mechanisms, achieving an F1 score of 96.27% on the Chinese-Old task, but its recall rate on the Chinese-Tai task (89.10%) still has room for improvement. The zero-shot method of hfl-llama3-it showed a sharp drop in recall to 81.30% on the Chinese-Thai task, while the recall was only 47.50% on the Chinese-Old task, reflecting that large models have limited transfer capabilities to low-resource languages ​​when there is a lack of targeted optimization.

[0138] Experiment 2: Ablation Experiment;

[0139] The impact of different modules on experimental results was verified on Chinese-Thai and Chinese-Lao parallel sentence pair extraction tasks.

[0140] Table 2 shows the comparison of the extraction and defusion experiments of parallel sentences between China and Thailand / China and Laos.

[0141]

[0142] Ablation experiments show that the Romanization feature, English-Lao-Thai trilingual transfer, and parallel sentence pair extraction supervision modules proposed in this invention all play a crucial role in the Chinese-Lao-Thai parallel sentence pair extraction task, but the contributions of different modules vary across different language pairs. In the Chinese-Thai task, removing the Romanization module (-Romanization) slightly increased the F1 score from 95.90% to 96.09%, but decreased the recall by 0.2 percentage points, indicating that while Thai Romanization can enhance cross-linguistic association, its filtering effect on semantic interference may affect the model's capture of specific linguistic features. When both Romanization and parallel sentence pair extraction modules are removed simultaneously (-Romanization-parallel sentence pair extraction), the F1 score drops sharply to 89.86%, indicating that Romanization features need to work in conjunction with explicit alignment supervision to effectively model cross-linguistic mapping. Removing the English-Lao-Thai trilingual module alone (-English-Lao-Thai) reduced the F1 score to 95.47%, while removing both the Romanization and English-Lao-Thai modules simultaneously (-Romanization-English-Lao-Thai) resulted in an F1 score of 95.36%, validating the auxiliary enhancement effect of the English bridging strategy on Thai representations. Furthermore, when the English-Lao-Thai trilingual alignment mechanism and the parallel sentence pair extraction module were removed (retaining only the Thai Romanization conversion task), the F1 score for the Chinese-Thai parallel sentence pair extraction task significantly decreased to 69.69%. Further research revealed that introducing auxiliary datasets unrelated to the main task triggers catastrophic forgetting in large language models, leading to a precipitous drop in performance on the parallel sentence pair extraction task.

[0143] In the Chinese-Lao task, removing the Romanization module (-Romanization) did not change the F1 score (99.25%), but increased the recall by 0.2 percentage points, indicating that Lao Romanization may introduce noise, and its shared character space has limited gain for semantic alignment. However, when the English-Lao-Thai trilingual module (-English-Lao-Thai trilingual module) was removed, the F1 score dropped significantly to 95.88%, and the recall decreased by 4.8 percentage points, indicating that English, as a high-resource language, plays a more critical role in the transfer of Lao language. Notably, removing the parallel sentence pair extraction module (-parallel sentence pair extraction) caused the F1 score of the Chinese-Lao task to plummet from 99.25% to 45.74%, with a recall of only 29.80%, proving that explicit alignment signals are indispensable for the supervision of low-resource language pairs. When both the Romanization and parallel sentence pair extraction modules were removed (-Romanization-parallel sentence pair extraction), the F1 score further decreased to 67.32%, reflecting a synergistic optimization effect between the two in the Lao task. The original model (hfl-llama3-it) has an F1 score of only 60.63% in the Chinese-Lao task, while the method of this invention improves it to 99.25% through module combination, which shows that multi-module joint optimization can overcome the inherent bottleneck of low-resource language alignment.

[0144] Experiment 3: Corpus Quality Comparison Experiment;

[0145] Table 3 Comparison of Corpus Quality

[0146]

[0147] The corpus quality experiments show that the new dataset constructed based on the method of this invention significantly improves the performance of the machine translation model. As shown in Table 3, after training with the newly constructed dataset, the BLEU score for the Lao-Chinese translation task increased from 15.51 in the original dataset to 17.35, and the BLEU score for the Thai-Chinese task achieved a significant increase, rising from 11.52 to 19.10. This improvement confirms that the semantic alignment quality of parallel sentence pairs in the newly constructed dataset is higher, verifying the effectiveness of the Romanization feature enhancement and multi-task joint training strategy for low-resource language alignment. In particular, in the Thai-Chinese task, the BLEU score increased by 65.8%, indicating that the semantic bias and noise problems existing in the original Thai data are significantly alleviated by the method of this invention. From the perspective of downstream translation tasks, this demonstrates the effectiveness of the method of this invention in improving the quality of low-resource corpora, providing high-quality data support for subsequent machine translation research.

[0148] Experiment 4: Case Analysis;

[0149] Table 4 presents examples of Thai-Chinese language exchanges.

[0150]

[0151] Table 5 presents an analysis of Lao-Chinese language examples.

[0152]

[0153] Case studies demonstrate that the method of this invention can effectively identify deep semantic differences, avoiding misjudgments caused by superficial lexical similarities in the original model. As shown in Tables 4 and 5, the Thai example sentence focuses on "social reflection," while the Chinese sentence emphasizes "technology application." The original model misjudged it as a parallel sentence pair due to capturing co-occurring words such as "artist," while the method of this invention identifies the topic shift through semantic associations of Romanized characters. The Lao sentence describes "the origin of the universe," while the Chinese sentence involves "future prediction." The original model incorrectly judged the sentence due to interference from the keyword "universe," while this method accurately captures the semantic gap between evolution and prediction using an English bridging strategy, verifying the effectiveness of cross-linguistic representation enhancement.

[0154] The specific embodiments of the present invention have been described in detail above with reference to the accompanying drawings. However, the present invention is not limited to the above embodiments. Within the scope of knowledge possessed by those skilled in the art, various changes can be made without departing from the spirit of the present invention.

Claims

1. A method for extracting parallel sentence pairs from a large model based on Romanization feature enhancement, characterized by: The method includes: Step 1: Data Preprocessing The texts of Thai and Lao were converted into Romanized characters using a Romanization conversion tool, and a Thai-Romanized and Lao-Romanized training instruction set was constructed. Based on open-source corpora and comparable Internet corpora, we constructed English-Thai and English-Lao bilingual translation instruction sets and Chinese-Thai and Chinese-Lao parallel sentence pair extraction instruction sets; Step 2: Train a large-scale parallel sentence pair extraction model based on Romanization feature enhancement, including: Step 2.1: Train the large language model to learn the Romanization knowledge of Tai Lao and enhance cross-language relevance by utilizing its shared character space; Step 2.2: With the help of a large-scale parallel corpus of English, the large language model learns the syntax-semantics mapping rules between English and Thai and Lao through joint training of "English→Thai" and "English→Lao" translation tasks, so as to enhance the alignment ability between Chinese and Thai and Lao. Step 2.3: Use Chinese-Thai and Chinese-Lao parallel sentences to perform supervised fine-tuning of the extracted instruction set, and optimize the target language semantic mapping capability of the model by explicitly aligning signals; Step 2.4 Finally, the training instruction set constructed in Step 1 is used to perform LoRA incremental fine-tuning using a multi-task joint training strategy; Step 3: Construct a large-scale parallel sentence pair extraction device based on Romanization feature enhancement: First, deploy the large-scale parallel sentence pair extraction model based on Romanization feature enhancement trained in Step 2 to the server and provide the corresponding service interface; then, construct a visualization platform for Chinese-Lao-Tai parallel sentence pair extraction; finally, obtain the corresponding parallel sentence pair extraction results by calling the model interface deployed on the server, thus forming a large-scale parallel sentence pair extraction device based on Romanization feature enhancement for parallel sentence pair extraction.

2. The method for extracting parallel sentence pairs from a large model based on Romanization feature enhancement as described in claim 1, characterized in that: Step 1 includes: Step 1.1, Data Construction: First, based on the Romanization conversion tool, the open-source corpus OPUS, and comparable corpora collected from the Internet, construct the instruction set required for multi-task joint training; Specifically, this includes: a training instruction set for Thai-Romanization and Lao-Romanization; a bilingual translation instruction set for English-Thai and English-Lao to support training for English-Thai and English-Lao translation tasks; and a parallel sentence pair extraction instruction set for Chinese-Thai and Chinese-Lao. Step 1.2, Data Processing: After the initial construction is completed, all corpora are processed uniformly, including deduplication to avoid duplicate samples interfering with the training effect; at the same time, samples containing missing values ​​are removed to ensure the integrity and consistency of the corpus. Step 1.3, Instruction Set Construction: Finally, the processed corpus is divided into training set, validation set and test set to form the instruction set required for multi-task joint training.

3. The method for extracting parallel sentence pairs from a large model based on Romanization feature enhancement as described in claim 1, characterized in that: The specific training process for the large language model in Step 2 includes: The input to a large language model that only decodes is a sequence. , m represents the length of the tokens in the input sequence, and each token Through word embedding matrix Convert to vector , with position encoding Overlay: ； in, Represents token The corresponding embedding vector; Indicates the vocabulary size of the model Indicates the length of the vector converted from each token; position encoding The calculation formula is: ； ； in, The positional encoding vector takes values ​​in even-numbered dimensions, represented by a sine function; The positional encoding vector takes values ​​in odd-numbered dimensions, represented by a cosine function; This represents the dimension index in the positional encoding vector; Obtain the initial embedding matrix The matrix is ​​input to the system by In an architecture where Transformer blocks are stacked, multi-head self-attention calculation of the mask is performed. Projected to Subspace: ； in, express The output of the layer Transformer block, , , Let represent the query, key, and value of the j-th projection subspace, respectively. , , Let represent the weights of the query, key, and value in the j-th projection subspace, respectively. , Indicates the number of attention heads in multi-head attention; the output of each single-head attention. for: ； in, Indicates the first A person's attention Matrix weights This indicates that the output of the attention head is Softmax normalized; mask matrix It is to ensure the position Accessible only The token is obtained by concatenating multiple outputs and then performing a linear transformation: ； in, This represents the final output of the attention layer. This indicates that the outputs of multi-head attention are spliced ​​together. Indicates the output projection matrix; performs residual connections and layer normalization: ； in, This represents the representation after performing the multi-head attention mechanism, followed by residual connections and layer normalization. This indicates that a layer normalization operation is being performed; The feedforward network (FNN) performs a nonlinear transformation on the normalized result: ； in, This represents the output after passing through the FFN layer; It is a commonly used nonlinear activation function; , This represents the two weight matrices of a feedforward network (FFN). , Represent the two bias vectors of the feedforward network FFN; The residuals are then reconnected and normalized to obtain the output of the current layer. : ； go through After layers are stacked, the final hidden state By outputting the projection matrix Convert to word probability distribution : ； Each location Word probability distribution for: ； in, Indicates a given Under the condition of model prediction The probability of; The model training uses weighted maximum likelihood estimation as the model's loss function. : ； in, Represents the input sequence; Represents the target sequence; Indicates the distribution of the training data; Represents the parameters of the model; This indicates that the model performs well under given input. and parameters Predict The conditional probability; Represents the input sequence The weighting function is used to adjust the importance of different samples; Represents the input sequence and target sequence The maximum likelihood.

4. The method for extracting parallel sentence pairs from a large model based on Romanization feature enhancement as described in claim 1, characterized in that: The specific steps of Step 3 include: Step 3.1: Deploy the large model using the Xinference open-source framework to the server. The trained parallel sentence pair extraction model based on Romanization feature enhancement is deployed to the server. Xinference provides a unified API interface to call the model. Step 3.2: Use Python's Streamlit framework to build a visualization platform for extracting parallel sentence pairs in Chinese, Lao, and Thai languages. Users input sentences to be compared in both languages ​​through the visualization interface, and send the input text information to the server through the API interface. Finally, the parallel sentence pair extraction model based on the Romanization feature enhancement performs inference to obtain the parallel sentence pair extraction results and returns them to the user.

5. A method and apparatus for extracting parallel sentence pairs from a large model based on Romanization feature enhancement, characterized in that: The apparatus includes a module for performing the large model parallel sentence pair extraction method based on Romanization feature enhancement as described in any one of claims 1 to 4.

6. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the large model parallel sentence pair extraction method based on Romanization feature enhancement as described in any one of claims 1 to 4.

7. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the large model parallel sentence pair extraction method based on Romanization feature enhancement as described in any one of claims 1 to 4.

8. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the large model parallel sentence pair extraction method based on Romanization feature enhancement as described in any one of claims 1 to 4.

Images