Low-resource language relation extraction data set construction method based on multilingual machine translation model
The construction of low-resource language relationship extraction data sets through multilingual machine translation models solved the problem of unsatisfactory relationship extraction effect in low-resource language environments, realized the construction of high-quality data sets, improved the model performance and application reliability, and promoted the Global information processing and language diversity protection.
Patent Information
- Application Number
- CN202411148768.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-21
- Publication Date
- 2024-12-03
AI Technical Summary
In a low-resource language environment, existing relationship extraction methods, especially the application of neural networks and pre-trained language models, face the problem of unsatisfactory results caused by data scarcity and differences in language characteristics, especially in Central Asia. In low-resource languages in Southeast Asia and the Middle East.
Using multilingual machine translation model to build a low-resource language relationship extraction data set, collect English relationship extraction corpus for preprocessing and translation, use NLLB model for translation, and use BERT as an evaluator to screen data to ensure data quality and form high-quality Low resource relationship extraction data sets.
It improves the relationship extraction performance in low-resource language environment, improves the generalization ability and application reliability of the model, and promotes the popularization of global information processing capabilities and language diversity protection.
Smart Images

Figure CN119067134A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to natural language processing, computational linguistics and relation extraction, and is a relation extraction dataset in a low-resource scenario, and more particularly to a method for constructing a low-resource language relation extraction dataset based on a multilingual machine translation model. Background Art
[0002] The rapid development of information technology in today's world has brought the era of big data to a climax, which has not only changed the way we process information and make decisions, but also promoted the innovation and application of natural language processing (NLP) technology. In the field of natural language processing (NLP), relation extraction (RE) as a core technology plays an important role in converting unstructured text into structured knowledge. Especially in the fields of knowledge graph construction, intelligent search and recommendation systems, the application of relation extraction (RE) technology plays a vital role in improving the intelligence of the system and user experience.
[0003] Traditional RE methods mainly rely on feature engineering and shallow learning models, such as support vector machines (SVM) and conditional random fields (CRF). These methods can achieve good results when there is sufficient data, but they often fail to realize their potential when faced with the problem of data scarcity in low-resource languages. With the rise of deep learning technology, neural network-based RE methods have gradually become mainstream. These methods can automatically learn semantic and structural information in text by utilizing model structures such as convolutional neural networks (CNN), recurrent neural networks (RNN), and attention mechanisms, thus overcoming the limitations of traditional methods to a certain extent.
[0004] In recent years, the rapid development of pretrained language models (PLMs) has promoted the advancement of relation extraction (RE) technology. Pretrained language models (PLMs) such as BERT (Bidirectional Encoder Representations from Transformers) and GPT (Generative Pretrained Transformer) are pre-trained on large-scale corpora, enabling the models to capture rich language expressions and semantic information. On this basis, fine-tuning these pretrained models to adapt them to specific RE tasks and low-resource language environments has become one of the effective strategies to improve relation extraction (RE) performance.
[0005] Although PLMs perform well in high-resource languages, they face many challenges in low-resource languages, such as lack of annotated data, differences in language characteristics, and limitations in model generalization capabilities. To address these issues, researchers are constantly exploring new methods such as multilingual data translation and cross-language transfer learning to improve RE performance and reliability in low-resource language environments.
[0006] This paper is dedicated to building and validating relation extraction datasets in low-resource languages, especially focusing on ten major low-resource languages in Central Asia, Southeast Asia, and the Middle East. Using advanced multilingual machine translation technology, publicly available English relation extraction (RE) datasets are translated into the target language, and data quality is screened through language perplexity (PPL) to ensure the validity and applicability of the data.
[0007] Through empirical research and comparative analysis, we evaluated the relationship extraction capabilities of various open source PLMs in low-resource language environments, demonstrating the potential of these technologies in promoting the popularization of global information processing capabilities and the protection of language diversity. This will not only help improve the information acquisition efficiency and experience quality of global users, but also open up new prospects and opportunities for academic research and commercial applications in related fields.
[0008] In summary, the research and application of low-resource language relation extraction technology is not only of great significance at the technical level, but also an important driving force for promoting the global development of information technology and the protection of cultural diversity. With the continuous advancement of technology and the increasing demand for global information exchange, we look forward to further promoting the development of relation extraction technology in low-resource language environments through interdisciplinary cooperation and innovation, and contributing more wisdom and strength to the sustainable development of the global information society.
[0009] The core technical problem of the present invention is to solve the challenge of relation extraction in low-resource language environments. Although existing relation extraction methods such as those based on neural networks and pre-trained language models (PLMs) have made significant progress, these models can automatically capture semantic and structural information in texts through pre-training on large-scale corpora, but their application still faces certain limitations in the face of data scarcity and language characteristics differences in low-resource languages. In particular, in the ten major low-resource languages in Central Asia, Southeast Asia and the Middle East, existing methods are difficult to achieve ideal relation extraction effects. Summary of the invention
[0010] The present invention aims to provide a method for constructing a low-resource language relation extraction data set based on a multilingual machine translation model. The method collects a high-resource (English) relation extraction data set, screens and preprocesses it, removes data whose length does not meet the requirements, and removes stop words; uses a translation system to translate the obtained data to obtain a relation extraction data set in a low-resource scenario, and adopts two methods for translation, one is to directly translate the high-resource data set, and the other is to first translate the original high-resource data set into an intermediate language, and then translate the intermediate language into a low-resource language; uses Bert as an evaluator to screen the generated data, filters out data whose perplexity is higher than a threshold, and then obtains the final data.
[0011] The method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model described in the present invention is carried out in the following steps:
[0012] a. Collect English relation extraction corpus and carefully select datasets from three main data sources: NYT10, FewRel and CrossRE corpora.
[0013] b. Perform data preprocessing and data cleaning on the monolingual corpus collected in step a: clean the collected corpus, remove HTML tags, special characters and irrelevant information, standardize the text format, use the preprocessing tools provided by NiuTrans to clean the data, eliminate illegal characters, and develop a series of preprocessing tools using Python to perform some operations, including secondary preprocessing, cleaning the data again, eliminating blank lines, and eliminating illegal characters;
[0014] c. Deploy the translation model and translate the original English relation extraction data using the NLLB model. Two methods are used for translation: one is to directly translate the high-resource dataset, and the other is to first translate the original high-resource dataset into an intermediate language and then translate the intermediate language into a low-resource language.
[0015] d. After obtaining the translated data set, use BERT as the evaluator to screen the generated data, use the evaluator to filter out the data with poor quality, and finally obtain the data with better quality to constitute the final data set.
[0016] The present invention provides a method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model. The method:
[0017] We collected English relation extraction corpora and carefully selected datasets from three main data sources: NYT10, FewRel, and CrossRE corpora, ensuring that the data came from public datasets or manually prepared data. To ensure the quality of the dataset, after obtaining the translated dataset, we used an evaluator to filter out data with poor quality, and finally obtained data with good quality to form the dataset. Furthermore, in order to fully verify the performance of the dataset on the currently popular large models, we selected 5 large models and tested them after simple fine-tuning to familiarize ourselves with the tasks, and obtained the test scores of the dataset on these large models. The quality of the dataset is a basic condition that must be met in model training. The dataset must come from a public high-quality dataset or manually prepared data.
[0018] Relation extraction collects text data from various sources including news, social media, academic papers, conversation records, etc. Sample quantity: In order to train a relation extraction model with high generalization ability, tens of thousands of labeled samples are usually required. The specific number should be adjusted according to the complexity of the model and the difficulty of the task. Data diversity: Collect corpora with diverse contexts and multiple relation types to improve the performance of the model in different scenarios;
[0019] Data accuracy: Ensuring the accuracy of labeled data is key. Annotation errors will directly affect the performance of the model. It is recommended that professional annotators perform annotation and conduct multiple rounds of review. Corpus representativeness: The selected corpus should be representative and cover common relationships and entity types in the target application scenario. Avoid corpus biased towards a specific field or type. Noise filtering: In the process of collecting corpus, noise needs to be filtered, for example, irrelevant content, advertisements, duplicate data, etc. need to be deleted to ensure the high quality of the corpus.
[0020] The collected monolingual corpus was preprocessed, and the selected data set was processed in detail. The preprocessing tools provided by NiuTrans were used to clean the data set and remove illegal characters. A series of Python preprocessing tools were used to further optimize the data processing process. These tools are not only used for secondary data cleaning, but also include removing blank lines, eliminating illegal character operations, removing HTML tags, special characters and irrelevant information, and standardizing text formats.
[0021] Deploy the translation model and translate the original English relation extraction data into low-resource relation extraction datasets in two ways; use the No Language Left Behind (NLLB) model for translation. NLLB is a multilingual translation project promoted by Meta (formerly Facebook) that aims to achieve high-quality translation of all languages in the world through advanced neural network technology. The core goal of the project is to ensure that all language users can enjoy efficient and accurate machine translation services regardless of the popularity of the language; use two methods to generate low-resource relation extraction datasets. One is to directly translate the English relation extraction dataset into the corresponding low-resource relation dataset, and the other is to first translate the English relation extraction dataset into an intermediate language dataset, and then translate it into the target low-resource language dataset;
[0022] The low-resource relationship extraction data set is screened using language perplexity to filter out data of low quality, and the remaining data set is stripped of data sets with relatively high perplexity to obtain the final data set.
[0023] Word segmentation and tagging: Perform word segmentation on the text to ensure that each word and punctuation mark is correctly separated. Then perform basic processing such as POS (part-of-speech tagging) and NER (named entity recognition) to provide a basis for relationship extraction. Dependency parsing: Perform dependency syntax parsing on the text to generate a dependency tree structure to capture the grammatical relations in the sentence and assist in the relationship extraction task. Data enhancement: Generate more training samples through data enhancement technology, such as synonym replacement, data back translation, etc., to improve the robustness of the model. Sample balance: During the processing, pay attention to balancing the number of samples of different relationship types to avoid too few samples of certain relationship types causing the model to be biased towards certain specific relationships.
[0024] The relationship extraction dataset in low-resource scenarios includes Uyghur (Ug), Kazakh (Kz), Kyrgyz (Ky), Tajik (Tg), Turkmen (Tk), Uzbek (Uz), Filipino (Fil), Indonesian (Id), Hebrew (He), and Persian (Fa), which belong to the fields of natural language processing and computational linguistics.
[0025] Language perplexity is one of the important indicators to measure the quality of text generated by language models. It is usually used to evaluate the naturalness and fluency of generated text, that is, the accuracy and coherence of the text in terms of grammar, semantics and logic. In natural language processing, language perplexity can be measured by calculating the probability that the model will generate the next word given a text sequence. Lower language perplexity usually means that the text generated by the model is more natural and fluent. For language generation tasks, especially in relation extraction or text generation, high-quality generated text needs to not only correctly express semantic relationships, but also avoid grammatical errors, ambiguity and incoherence. Therefore, reducing language perplexity is one of the key steps to improve the quality and application practicality of text generated by language models. In practical applications, researchers and developers usually use language perplexity as an important criterion for evaluating model improvements and optimizations to ensure that the quality of generated text reaches the expected level and meets the requirements of specific tasks.
[0026] The present invention discloses a method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model. The information of the large model used for test data in the method is as follows:
[0027] LLaMA (Large Language Model for AI) is a large-scale language model launched by Meta (formerly Facebook) to promote cutting-edge research in natural language processing (NLP) and generation tasks; LLaMA uses the Transformer architecture, which is highly scalable and flexible and can handle a variety of tasks from text generation, text classification to question-answering systems. Its training data covers a wide range of corpora, including online articles, books, and Wikipedia, thus ensuring the diversity and generalization of the model. LLaMA performs particularly well in handling multilingual translation and low-resource language tasks;
[0028] Falcon (NVIDIA Falcon) is a large-scale language model developed by NVIDIA, focusing on high-performance computing and deep learning applications. Falcon uses NVIDIA's GPU acceleration technology and Transformer architecture to efficiently process massive data and complex tasks. The model performs well in multiple NLP benchmarks and is suitable for a variety of application scenarios such as natural language generation, machine translation, text summarization, and sentiment analysis. Falcon's training relies on huge computing resources and optimized training algorithms, ensuring its leading position in performance and accuracy;
[0029] BLOOMZ (BigScience Large Open Multilingual Model Zoo) is an open-source large-scale language model developed by EleutherAI to promote open science and democratize AI research. BLOOMZ's training data covers a wide range of Internet corpora, which are carefully cleaned and filtered to ensure data diversity and quality. The model uses the GPT architecture and is capable of performing multiple tasks such as natural language generation, text understanding, and language translation. The open source nature of BLOOMZ allows researchers and developers to freely access, use, and improve the model, thereby promoting the popularization and development of AI technology;
[0030] Qwen (Quantum Well-Established Natural Language Processing Model) is a large-scale language model developed by Alibaba DAMO Academy, focusing on promoting innovation in the fields of natural language processing and human-computer interaction. Qwen is trained using massive Chinese and multilingual corpora and has powerful text generation and comprehension capabilities. The model uses an advanced Transformer architecture and performs well in a variety of NLP tasks, such as machine translation, text summarization, dialogue systems, and knowledge graph construction. Qwen is designed to provide efficient, accurate, and natural language processing solutions to help all industries upgrade to intelligence;
[0031] OLMo (Open Language Model) is a large-scale language model developed by OpenAI, dedicated to advancing the frontier of natural language processing and generation technology. Based on the GPT architecture, OLMo has excellent text generation, understanding and reasoning capabilities after training with massive data. The model performs well in various NLP tasks, including question-answering systems, machine translation, text classification and sentiment analysis.
[0032] The present invention discloses a method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model. The method proposes a relationship extraction dataset in a low-resource scenario, which can well promote the development of the relationship extraction field in low-resource scenarios.
[0033] Scoring metrics are used to evaluate the performance of the models; experimental results show that LLaMA and Falcon perform significantly better, which can be attributed to their powerful pre-training mechanisms and ability to adapt to the nuances of low-resource languages. BRIEF DESCRIPTION OF THE DRAWINGS
[0034] Figure 1 It is a flow chart of the present invention;
[0035] Figure 2It is a model flow chart of the present invention for constructing a low-resource relationship extraction data set through a high-resource relationship extraction data set;
[0036] Figure 3 These are two methods (approaches) for constructing low-resource datasets proposed in the present invention;
[0037] Figure 4 This is a comparison chart of model complexity of two methods for constructing low-resource datasets. DETAILED DESCRIPTION
[0038] The following is a further detailed description of the specific implementation of the present invention in conjunction with the accompanying drawings and examples. The following examples are only used to illustrate the present invention, but are not limited to the scope of the present invention.
[0039] Example 1
[0040] The method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model described in the present invention is carried out in the following steps:
[0041] a. Collect English relation extraction corpus and carefully select datasets from three main data sources: NYT10, FewRel and CrossRE corpora.
[0042] b. Perform data preprocessing and data cleaning on the monolingual corpus collected in step a: clean the collected corpus, remove HTML tags, special characters and irrelevant information, standardize the text format, use the preprocessing tools provided by NiuTrans to clean the data, eliminate illegal characters, and develop a series of preprocessing tools using Python to perform some operations, including secondary preprocessing, cleaning the data again, eliminating blank lines, and eliminating illegal characters;
[0043] c. Deploy the translation model and translate the original English relation extraction data using the NLLB model. Two methods are used for translation: one is to directly translate the high-resource dataset, and the other is to first translate the original high-resource dataset into an intermediate language and then translate the intermediate language into a low-resource language.
[0044] d. After obtaining the translated data set, use Bert as the evaluator to filter the generated data, use the evaluator to filter out the data with poor quality, and finally obtain the data with better quality to form the final data set;
[0045] Figure 2 It shows a process of converting a high-resource relation extraction dataset into a low-resource relation extraction dataset:
[0046] First, we have a high-resource relation extraction dataset, which contains original sentences, head entities, tail entities, and the relationships between them. In order to construct a low-resource relation extraction dataset, we use translation software to translate these data into corresponding low-resource language data. This step ensures that we have the same sentences and relationship information as the high-resource dataset in the low-resource language environment.
[0047] Next, the translated low-resource dataset is processed and cleaned. Specifically, a filter is used to filter out unqualified data whose perplexity (i.e., translation quality indicator) is greater than a certain threshold. The greater the perplexity, the lower the translation quality, so a threshold is set to ensure the quality of the data.
[0048] This method can retain data with high translation quality, thus obtaining the final low-resource relationship extraction dataset. This filtered and cleaned dataset is more accurate and reliable, which helps improve the results of subsequent model training and evaluation. This not only enables relationship extraction tasks in low-resource language environments, but also ensures data quality, thereby improving model performance and reliability in practical applications.
[0049] Figure 3 Two methods for obtaining low-resource relation extraction datasets from high-resource relation extraction datasets are shown. Figure 4 Comparison chart of model complexity of two methods for constructing low-resource datasets. Use the translation model to translate the original English relation extraction dataset into the corresponding low-resource relation extraction dataset, and then use language perplexity to filter out low-quality datasets with high perplexity, thereby obtaining a high-quality dataset.
[0050] There are two ways to obtain the data set. One is to preprocess, filter out data with length <10, >50, and duplicate data, directly translate it into the corresponding low-resource data set through the translation system, and then filter it through the evaluator;
[0051] The second method is to use a translation system to translate the data into the corresponding intermediate language dataset (such as Chinese) after preprocessing, and then translate the intermediate language dataset into the corresponding low-resource dataset. The evaluator then filters out the data with a perplexity greater than the threshold, and the remaining data is the final dataset.
[0052] In order to verify the effectiveness of the filter, the perplexity of the data before and after filtering was tested. The results are shown in Table 1. It can be seen that the perplexity of the data after filtering has dropped a lot, proving the effectiveness of the filter;
[0053] Table 1. Comparison of perplexity before and after filtering for each language
[0054] language Unfiltered Filtered △PPL (text complexity) KZ 15.37 7.27 8.10 Ky 7.29 7.10 0.19 Tg 9.09 8.02 1.08 Uz 23.99 19.50 4.49 Id 30.96 15.41 15.55 He 11.42 9.92 1.51 Fa 37.05 17.41 19.64 Kz-p2 10.12 7.11 3.02
[0055] The perplexity of datasets generated by different methods is also tested, such as Figure 1 As shown, it can be seen that the perplexity of translating directly from English to the target language is higher than that of translating English into an intermediate language and then into the target language, proving that the second method has higher fluency and quality of the dataset.
[0056] The performance of the relation extraction dataset on different large models was tested. The test results are shown in Table 2.
[0057] Table 1 Test scores of the constructed dataset on the large model
[0058]
[0059] From the results in Table 2, we can see that LlaMA performs relatively well in data sets of various languages. This is because the LlaMA large model has strong multilingual capabilities and semantic understanding capabilities, which can better understand the meaning of the task and make better responses on low resources. Qwen also performs relatively well in low-resource languages, which may be because the pre-training data set of the Qwen large model is relatively balanced, while other large models do not perform very well in low-resource scenarios, which may be caused by imbalanced training data.
[0060] The invention discloses a method for constructing a low-resource language relation extraction dataset based on a multilingual machine translation model. The creation of the dataset provides a new approach to the field of relation extraction in low-resource languages (such as Uyghur), and also makes many contributions to academia and industry. First, by providing a reliable relation extraction dataset in a low-resource scenario, a stable foundation is provided for researchers to develop and evaluate new relation extraction algorithms and models. This not only helps to promote research progress in the fields of natural language processing and computational linguistics, but also promotes the development of multilingual information processing technology. Secondly, from the perspective of industrial applications, the dataset provides support for various language processing applications. In a multilingual environment, especially in the context of low-resource languages, relation extraction is a key component of many intelligent applications (such as information retrieval, intelligent question-answering systems, etc.). By providing such a dataset, it can help enterprises and developers build more accurate and reliable multilingual processing tools, thereby improving user experience and service quality. In short, this dataset not only contributes to the in-depth exploration of academic research, but also directly promotes technological progress and innovation in practical applications.
Claims
1. A method for constructing a low-resource language relationship extraction dataset based on a multilingual machine translation model. It is characterized in that Follow these steps: a. Collect English relation extraction corpus and carefully select datasets from three major data sources: NYT10, FewRel and CrossRE corpora. b. Perform data preprocessing and data cleaning on the monolingual corpus collected in step a: clean the collected corpus, remove HTML tags, special characters and irrelevant information, standardize the text format, use the preprocessing tools provided by NiuTrans to clean the data, eliminate illegal characters, and develop a series of preprocessing tools using Python to perform some operations, including secondary preprocessing, cleaning the data again, eliminating blank lines, and eliminating illegal characters; c. Deploy the translation model and translate the original English relation extraction data using the NLLB model. Two methods are used for translation: one is to directly translate the high-resource dataset, and the other is to first translate the original high-resource dataset into an intermediate language and then translate the intermediate language into a low-resource language. d. After obtaining the translated data set, use BERT as the evaluator to screen the generated data, use the evaluator to filter out the data with poor quality, and finally obtain the data with better quality to constitute the final data set.
