Method for improving and constructing quality of low-resource multilingual supervised dataset

Abstract

The application discloses a low-resource multilingual supervised data set quality improvement and construction method, belonging to the technical field of natural language processing and data construction, comprising the following steps: defining a task system and collecting original data; automatically obtaining the original data and parsing them into standardized texts; a semantic cleaning and privacy protection preprocessing stage for deeply cleaning the texts, segmenting semantic blocks and desensitizing private information; a structured encapsulation and man-machine collaborative labeling stage for converting the cleaned texts into high-quality instruction fine-tuning data; a quality verification and data set solidification stage for quality verification and solidification of the instruction fine-tuning data set; and a data asset management and continuous evolution stage for safe management, version control and continuous iteration of the data assets. The method significantly improves the alignment quality and robustness of the model in a low-resource multilingual environment, and provides a solid technical foundation for constructing high-performance multilingual LLMs.

Description

Technical Field

[0001] This invention belongs to the field of natural language processing and data construction technology, and relates to a method for improving the quality and constructing low-resource multilingual supervised datasets. Background Technology

[0002] With the rapid development of large-scale models and the pursuit of stronger multilingual understanding capabilities, building high-quality, large-scale multilingual datasets, especially supervised datasets for low-resource languages, has become a critical and challenging area.

[0003] Existing solutions:

[0004] (1) Professional human translation to build a high-quality dataset:

[0005] This approach selects sentences from trusted sources such as English Wikipedia and has them translated into multiple languages ​​by professional translators to build a high-quality evaluation benchmark. Its core lies in a rigorous quality control process involving multiple rounds of automated and manual checks to ensure the accuracy and naturalness of the translations. A typical example is the Flores-101 dataset, which professionally translates 3001 English sentences into 101 languages, providing a reliable evaluation basis for many-to-many machine translation systems.

[0006] (2) Creating parallel data by translating high-resource language datasets:

[0007] This method rapidly and cost-effectively creates cross-lingual parallel evaluation data by translating existing labeled datasets (such as question-answering and natural language inference datasets) in high-resource languages ​​like English into multiple target languages. This approach quickly expands the language coverage of benchmarks, and widely used cross-lingual understanding benchmarks such as XNLI and XQuAD are built using this method.

[0008] (3) Distant Supervision:

[0009] Automatic labeling is a low-cost method for generating training data that automatically assigns labels to unlabeled text using external knowledge sources such as knowledge bases, dictionaries, or predefined rules. This method is particularly suitable for information extraction tasks, such as automatically identifying place names by matching text with a place name dictionary, thus saving significant manual labeling costs.

[0010] (4) Cross-lingual projection:

[0011] This technique uses parallel corpora (i.e., bilingual aligned texts) as a bridge to project tags (such as parts of speech and syntactic structure) from high-resource language (e.g., English) sentences onto corresponding parts of aligned low-resource language sentences. This method is highly dependent on auxiliary resources such as high-quality parallel corpora, machine translation systems, or bilingual dictionaries.

[0012] (5) Crowdsourcing:

[0013] Crowdsourced annotation recruits a large number of workers to annotate data through online platforms (such as MTurk and Prolific) to achieve large-scale data collection. However, for low-resource languages, this approach faces a core challenge: how to ensure that annotators have sufficient language proficiency and effectively prevent them from cheating by using machine translation. This requires the design of strict quality control mechanisms, such as translation test questions.

[0014] (6) Data Augmentation:

[0015] Data augmentation techniques aim to artificially generate new training samples based on a limited amount of existing labeled data through a series of transformations. Common methods include back-translation (translating sentences into another language and then back), synonym substitution, and sentence structure transformation, thereby increasing the diversity of the data and improving the robustness and generalization ability of the model in low-resource scenarios.

[0016] (7) Use pre-trained multilingual models:

[0017] This approach directly utilizes language models (such as mBERT and XLM-R) pre-trained on large-scale multilingual corpora to transfer knowledge from high-resource languages ​​to low-resource languages ​​through zero-shot or few-shot learning. Its effectiveness depends on the target language having sufficient coverage in the model's pre-training corpus; otherwise, the transfer effect will be significantly reduced.

[0018] Problems with existing technology:

[0019] (1) High-quality data construction is costly and has poor scalability:

[0020] The fundamental challenge in building high-quality low-resource language datasets lies in their high cost and limited scalability. Organizing large-scale, professional human translation efforts is already extremely difficult due to the scarcity of skilled translators for low-resource languages. More importantly, ensuring data quality requires designing and implementing complex, multi-round quality control processes, including automated checks (such as preventing machine translation plagiarism) and independent human evaluation. This time-consuming and labor-intensive process severely limits the ability to scale this approach to more languages.

[0021] (2) Issues with data quality and authenticity are prominent:

[0022] Under existing construction methods, data quality and authenticity suffer from significant deficiencies. First, datasets created through translation commonly suffer from "translation artifacts," meaning the translations are unnatural, overly literal, and fail to accurately reflect the language habits and cultural background of native speakers. Second, the labels for many low-resource language datasets are automatically generated, lacking rigorous human verification and containing significant noise, severely impacting model training performance. Finally, many datasets have incomplete documentation, lacking crucial information about the construction process, annotation guidelines, and annotator backgrounds, making it difficult for researchers to accurately assess the dataset's reliability, bias, and applicability.

[0023] (3) Insufficient resource dependence and ecological support:

[0024] The development of low-resource NLP is severely constrained by a lack of basic resources and weak ecosystem support. On the one hand, advanced modeling techniques such as multilingual pre-trained models (mBERT, XLM-R) have insufficient coverage of low-resource languages, especially languages ​​from Africa and the Americas, resulting in limited transfer learning effectiveness. On the other hand, the available raw text data sources for low-resource languages ​​are also very limited, mostly confined to Wikipedia and news texts, lacking diverse corpora created for specific tasks. In addition, mainstream crowdsourcing platforms (such as MTurk) have very limited support for non-English languages, and their built-in language qualification certifications cover few languages, making it difficult to recruit qualified bilingual workers and effectively prevent them from cheating using machine translation, making it extremely difficult to obtain high-quality data through crowdsourcing.

[0025] In summary, current methods for training large language models for low-resource languages ​​still face several core challenges in data construction and capability assessment. These are mainly reflected in the following three aspects. First, existing methods rely on limited translation corpora and automatically labeled data, lacking high-quality, multi-dimensional capability training samples. Due to the significant differences in grammatical structure, cultural expression, and pragmatic habits among low-resource languages, traditional data construction methods based on machine translation or rule mapping struggle to accurately reflect the real-world usage scenarios and cognitive complexity of the language. This results in insufficient naturalness of the generated samples and weak generalization ability of the model in real-world applications. Second, existing training data often focuses on single tasks or shallow language comprehension, lacking systematic modeling of multi-level cognitive abilities. Higher-order cognitive abilities, from memory and comprehension to application, analysis, evaluation, and creation, are not organically connected within the current training framework, leading to uneven development of model capabilities and difficulty in handling real-world tasks requiring complex reasoning and creative thinking. Third, traditional data construction methods usually only focus on text content and ignore the laws of cognitive development. The training samples lack targeted design for metacognitive ability and long text generation ability, which cannot effectively support the model's self-evaluation and continuous optimization in low-resource language environment. Moreover, the training process is uncontrollable and it is difficult to achieve a step-by-step improvement of cognitive ability. Summary of the Invention

[0026] The purpose of this invention is to overcome the shortcomings of the aforementioned background technologies and provide a method for improving and constructing quality-enhancing multilingual supervised datasets for low-resource languages. This method provides a large-scale, high-quality, fine-grained, and broadly covered multilingual human feedback dataset (MRLHF) for low-resource languages ​​and its accompanying evaluation benchmark. Its specific objectives include: 1) Solving the data scarcity problem: Constructing a large-scale dataset covering 50 languages ​​(with a particular focus on low-resource languages) to provide a solid data foundation for training multilingual reward models and break the technological monopoly of high-resource languages. 2) Enhancing task cognitive depth: Based on Bloom's taxonomy of cognition, designing multi-dimensional tasks covering six cognitive levels from "memory" to "creation" (such as open-ended question answering, long text summarization, multi-turn dialogue, text completion, heuristic generation, etc.) to systematically improve the model's understanding, reasoning, and creative abilities. 3) Ensuring data authenticity and usability: Through the principles of "multi-source acquisition" and "real-world scenarios," data is obtained from internet forums, professional domain documents, and user content from multiple platforms to avoid synthetic data bias and ensure that the knowledge learned by the model can be applied to the real world. 4) Establish a comprehensive evaluation system: Create multi-dimensional benchmarks including CBRM-RewardBench (reward model evaluation), CBRM-Bench (multi-domain reasoning and generation evaluation), and CBRM-SafetyBench (safety evaluation) to provide standardized and reproducible evaluation criteria for the capabilities and alignment effects of multilingual models. 5) Build a scalable data production pipeline: Design a six-stage data construction framework encompassing goal definition, data collection and cleaning, formatted annotation, quality assurance, and iterative release to ensure high efficiency, high reliability, and sustainable evolution of data production, supporting the co-evolution of data and models.

[0027] The technical solution of this invention includes:

[0028] A method for improving and constructing quality of low-resource, multilingual supervised datasets includes:

[0029] S1, the task specification and multi-source acquisition phase, is used to define the task system and collect raw data;

[0030] S2, the automated data acquisition and multimodal document parsing stage, is used to automatically acquire raw data and parse it into standardized text;

[0031] S3, the semantic cleaning and privacy protection preprocessing stage, is used for deep cleaning of text, segmentation of semantic blocks and desensitization of privacy information;

[0032] S4, the structured encapsulation and human-machine collaborative annotation stage, is used to transform the cleaned text into high-quality instruction fine-tuning data;

[0033] S5, the quality verification and dataset solidification stage, is used to perform quality verification and solidification on the instruction fine-tuning dataset output by S4.

[0034] S6, the data asset management and continuous evolution stage, is used for the security management, version control and continuous iteration of data assets.

[0035] Beneficial effects:

[0036] Based on relevant research results and experimental verification, the framework of this invention significantly improves low-resource multilingual reasoning ability by 11% and reduces unsafe behaviors by 57% under the same training data scale. When applied to a 7B-scale model, its performance on open reasoning tasks surpasses larger instruction tuning systems (such as Qwen2.5-72B-Instruct and DeepSeek-R1-Distill-Qwen-32B) by up to 8%. These results demonstrate that the method of this invention significantly improves the alignment quality and robustness of the model in low-resource multilingual environments, providing a solid technical foundation for building high-performance, multilingual LLMs. Attached Figure Description

[0037] Figure 1 The diagram illustrates the technical block diagram of the present invention for improving and constructing a low-resource, multilingual supervised dataset. Detailed Implementation

[0038] Exemplary embodiments of the present disclosure will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the disclosure to those skilled in the art.

[0039] To achieve the aforementioned objectives, this invention proposes a cognitively oriented and quality-closed-loop driven multilingual low-resource data production method. This method, with "task definition first, quality throughout," differs from traditional, linear data collection approaches. Traditional methods typically prioritize data availability, stacking data from the bottom up, leading to a disconnect between data and training objectives, and quality relying on later sampling checks. In contrast, the systematic method of this invention starts with a clear model capability objective, implementing positive intervention at every stage of data production through structured task design and full-process quality control, forming a closed-loop control path of "objective definition → multi-source collection → cleaning and desensitization → formatted annotation → quality verification → version iteration."

[0040] The system focuses on five core tasks: open-ended question answering, long text summarization, multi-turn dialogue, text continuation, and heuristic generation. Strictly adhering to Bloom's Taxonomy, it reverse-engineers data specifications and sources, enabling the large-scale construction of high-quality, fine-tuned instruction data in low-resource, multi-language environments. This invention integrates advanced concepts from modern software engineering and data science (including API-driven batch data acquisition, semantic-based text segmentation, and multi-turn manual cross-validation) to construct a comprehensive data production system encompassing task planning, data acquisition, preprocessing, annotation and synthesis, and lifecycle management. This provides a reliable data foundation for cultivating the accuracy of low-resource, large-language models.

[0041] The system of this invention consists of six sequentially connected, iteratively feedback-based core stages: the goal and rule definition stage, the data collection and organization stage, the data cleaning and preprocessing stage, the data formatting and annotation stage, the quality assurance and dataset construction stage, and the security, maintenance, and iteration stage. The entire method follows a technical roadmap of "goal-oriented—multi-source acquisition—deep cleaning—standardized annotation—quantitative evaluation—continuous iteration," with each stage interconnected and operating collaboratively to form a closed-loop data production system driven by model capability requirements.

[0042] Figure 1 This diagram illustrates the technical block diagram of the method for improving and constructing low-resource, multilingual supervised datasets according to the present invention. Figure 1 As shown, the method includes the following steps:

[0043] S1. Task Specification and Multi-Source Acquisition Phase: This phase defines the task structure and collects raw data. This step includes the following sub-steps:

[0044] S1.1. Definition and Specification of Multi-Task Text System:

[0045] To systematically cultivate the comprehensive capabilities of Large Language Models (LLMs) in low-resource, multilingual environments, this invention first constructs a multi-task text system based on cognitive hierarchy theory. This system clearly defines five core task types and their capability cultivation objectives, serving as the top-level design for the entire data engineering process.

[0046] (1) Open-ended question answering: This aims to train the model to generate long texts and detailed factual content for open-domain questions, which is different from closed-ended multiple-choice questions, in order to strengthen the model's semantic memory and knowledge integration capabilities.

[0047] (2) Long text summarization: By requiring the model to condense information from long documents (5,000 to 10,000 words), the aim is to challenge and improve its deep understanding and core information extraction capabilities.

[0048] (3) Multi-turn dialogue: Simulates the interaction scenarios between real users and AI, covering daily communication, task-oriented dialogue and role-playing, aiming to enhance the model's ability to maintain context and follow instructions.

[0049] (4) Text continuation: Based on the sequence generation nature of LLMs, three structured subclasses, continuation, imitation and style transfer, were designed to systematically evaluate and improve the model’s ability to generate long texts coherently and adapt to style.

[0050] (5) Heuristic text generation: The model is required to generate complete and original text content based on open prompts, aiming to stimulate its imagination, context adaptability and generation diversity.

[0051] The task specification system directly determines the microstructure of the data samples. For example, the summarization task corresponds to the "document-summary" pair, and the question-answering task corresponds to the "question-answer" pair, providing inviolable structured constraints for subsequent data collection, cleaning, and labeling.

[0052] S1.2. Implement multi-source heterogeneous data acquisition:

[0053] Based on the aforementioned task system specifications, this invention implements a systematic data collection strategy for multilingual languages, particularly low-resource languages. The system aggregates large-scale, multi-source heterogeneous data from the internet, collaboration platforms, and structured documents, with data sources including Confluence, Notion, SharePoint, Google Docs, and various HTML pages. To ensure the timeliness and quality of the corpus, the system strictly limits the data collection timeframe to the second half of 2010 to 2025 and incorporates a built-in filtering mechanism to automatically exclude outdated, sensitive, and low-quality documents.

[0054] S1.3. Embedding a security and privacy compliance framework:

[0055] From the outset of data collection, the system incorporates a pre-defined security and privacy compliance framework. This framework establishes clear rules for identifying and de-identifying sensitive information based on the characteristics of text in different languages. Through regular expressions and Named Entity Recognition (NER) technology, it automatically locates and replaces personal identification information such as phone numbers, email addresses, and ID numbers. Simultaneously, the system enforces strict access control policies, ensuring through authentication and access logs that all data ultimately entering the dataset does not contain any unauthorized content, thus guaranteeing the legality and compliance of data use from the source.

[0056] S2. Automated Data Acquisition and Multimodal Document Parsing Stage: This stage automatically acquires raw data and parses it into standardized text. This step may include the following sub-steps:

[0057] S2.1. Building an automated data collection and metadata management engine:

[0058] This phase involves building an automated, traceable, large-scale data acquisition engine. This engine achieves large-scale, permission-based acquisition of raw documents by calling standardized application programming interfaces (APIs) provided by various platforms (such as the Confluence API and Google Drive API) or performing compliant database export operations (such as batch exporting in CSV format).

[0059] Unlike simple data scraping, the core feature of this engine lies in its embedded metadata management mechanism. During the collection process, the system automatically extracts and persistently stores key metadata related to document source and quality, including but not limited to: document creation date, collector identity, document type, and original Uniform Resource Locator (URL). This metadata layer provides a structured information foundation for the end-to-end traceability of data assets, subsequent time-series filtering, and data augmentation operations.

[0060] S2.2. Multimodal document parsing and intermediate representation generation:

[0061] To address the processing complexity arising from heterogeneous raw data formats, this invention designs a unified multimodal document parsing and conversion pipeline. This system integrates and schedules a dedicated parsing toolchain for mainstream document formats (such as .docx, .pdf, .pptx, .html, and .txt), uniformly converting heterogeneous documents into standardized plain text formats to form the "intermediate representation" required for downstream processing. The specific mapping relationship is as follows:

[0062] (1) PDF documents: PyMuPDF, pdfplumber or Adobe Acrobat are used to parse the documents and extract the text flow and layout information.

[0063] (2) DOCX document: Access the document object model through the python-docx library to extract structured text content such as paragraphs and tables.

[0064] (3) PPTX document: Using the python-pptx library, focus on extracting text box elements in the slides and retain the core narrative logic.

[0065] (4) HTML page: Use the BeautifulSoup library to parse markup language, strip HTML tags, and retain the main content text and necessary hierarchical structure.

[0066] After this stage of processing, the output is a preliminarily normalized plain text stream. It should be noted that this text stream still contains noisy elements unrelated to semantic understanding, such as headers, footers, page numbers, and code blocks. These elements will be selectively removed in the next stage of deep cleaning. The output of this stage provides a standardized and clearly structured input foundation for subsequent high-quality semantic processing and instruction fine-tuning tasks.

[0067] S3. Semantic Cleaning and Privacy Protection Preprocessing Stage: This stage involves deep cleaning of the text, segmenting semantic blocks, and de-identifying privacy information. This step may include the following sub-steps:

[0068] S3.1. Deploy a multi-level noise cleanup and structured information extraction engine:

[0069] This stage aims to perform in-depth cleaning and structural enhancement of the parsed raw text stream. The system deploys a multi-level noise removal and structured information extraction engine. Through pattern matching and a rule base, it automatically identifies and removes irrelevant text elements that interfere with semantic understanding, including but not limited to: headers and footers, page numbers, legal disclaimers, and duplicate content. Simultaneously, the engine performs character-level normalization, cleaning up redundant line breaks and broken spaces. Garbled characters such as incorrect UTF-8 encoding.

[0070] For non-plain text elements in the document, the engine integrates a dedicated processing module:

[0071] (1) Table structure conversion module: Extracts the identified table data and converts it into structured text formats such as Markdown to preserve its row and column relationships and data semantics.

[0072] (2) Image Optical Character Recognition (OCR) Module: This module performs evaluative extraction of text content embedded in images. The system employs a cost-benefit evaluation strategy, triggering the OCR process only when the value of the text information exceeds the computational cost, thus ensuring the effectiveness of resource investment.

[0073] S3.2. Execute a context-aware semantic segmentation algorithm to segment the text.

[0074] To overcome the inherent context window limitations of large language models, this invention proposes a context-aware semantic segmentation algorithm. This algorithm abandons simple fixed-length segmentation or coarse-grained segmentation methods based on newlines and periods, instead employing an advanced recursive character text splitter. Its core lies in the algorithm's proactive identification of semantically complete boundaries in the text (such as paragraph ends and chapter titles), prioritizing segmentation at these boundaries. To prevent semantic breaks in the context caused by segmentation, the algorithm introduces a controllable overlap between adjacent text blocks, thereby maintaining the coherence and integrity of the long document context during model training.

[0075] S3.3. Construct a text modality privacy-sensitive information desensitization system to replace the original sensitive information with standardized placeholders.

[0076] To strictly comply with data privacy regulations, a text-based privacy-sensitive information de-identification system was built. This system employs a two-layer recognition architecture:

[0077] (1) Pattern matching layer: Based on predefined regular expressions, it can accurately match sensitive information with fixed patterns, such as email addresses and phone numbers.

[0078] (2) Entity recognition layer: Using pre-trained natural language processing models (such as spaCy's NER model), it identifies and locates named entities such as people's names, place names, and organization names in unstructured text.

[0079] All identified sensitive information is systematically replaced with uniform, meaningless standardized placeholders (such as [PHONE], [NAME], [EMAIL]) to ensure that privacy risks are eliminated without compromising the grammatical structure and readability of the text.

[0080] S3.4. Redundancy Control and Data Reduction Processing:

[0081] To improve dataset quality and training efficiency, the system implements strict data redundancy control. This process consists of two steps:

[0082] (1) Precise deduplication: Delete duplicate paragraphs with exactly the same content.

[0083] (2) Fuzzy deduplication: SimHash and other fuzzy fingerprint algorithms are used to calculate the semantic similarity between texts and set a threshold to delete highly similar redundant paragraphs, thereby achieving the intensive use of the dataset while preserving data diversity and avoiding overfitting during model training.

[0084] S4. The structured encapsulation and human-machine collaborative annotation stage is used to transform the cleaned text into high-quality instruction fine-tuning data. This step may include the following sub-steps:

[0085] S4.1. Design a unified instruction fine-tuning data architecture to map diverse task types into a machine-readable structured format.

[0086] To adapt to the instruction fine-tuning paradigm of large language models, this phase defines and implements a unified and serializable instruction fine-tuning data architecture. This architecture uses the "instruction-input-output" triple as its core, mapping diverse task types into a machine-readable structured format. The specific encapsulation specifications are as follows:

[0087] (1) Single-round task encapsulation: For tasks such as question answering and summarizing, a JSON object structure of {"instruction": "...", "input": "...", "output": "..."} is adopted. Among them, the instruction field defines the task instruction, the input field carries the task input (such as a long document), and the output field stores the expected ideal output.

[0088] (2) Multi-turn dialogue encapsulation: For dialogue tasks, a message list-based serialization structure is adopted: {"messages": [{"role": "user", "content": "..."}, {"role": "assistant", "content": "..."}]}, in order to fully preserve the interaction context.

[0089] This architecture ensures the uniformity of data formats across different tasks, providing the model with clear and consistent learning signals, and forms the technical foundation for achieving efficient instruction fine-tuning.

[0090] S4.2. Based on the data architecture designed in S4.1, a hybrid annotation pipeline with human-machine collaboration is constructed.

[0091] To address the core challenge of the scarcity of high-quality labeled data for low-resource languages, this invention designs a three-path hybrid labeling pipeline that integrates the wisdom of human experts with the capabilities of automated models.

[0092] (1) Path 1: Expert writing and cross-validation mechanism.

[0093] For languages ​​with limited resources, the system recruits native speakers with master's degrees or higher to write the data. To ensure the absolute quality of the samples, the system enforces a cross-validation protocol of "one person annotating, four people quality checking." Only samples that have passed the independent review of all quality checkers can be included in the final dataset. This mechanism sets an extremely high quality floor for the core samples.

[0094] (2) Path 2: Model-assisted generation and manual correction loop.

[0095] The system integrates cutting-edge large language models (such as GPT-5 and Gemini-2.5-Pro) as annotation assistants, automatically generating question-answer pairs, abstract candidates, and other data based on cleaned text in S3. The model-assisted generation and human correction loop adopts an optimized "generation-review-correction" loop: the model is responsible for large-scale initial screening and generation, while human experts focus on quality review and fine-tuning of the generated results, achieving a balance between data production efficiency and quality.

[0096] (3) Path 3: Structured transformation of authoritative resources.

[0097] The system collaborates with experts and institutions in the field of linguistics to transform authoritative existing resources such as textbooks and examination materials into high-quality training samples that meet the requirements of instruction fine-tuning through a legal and standardized process, thereby enriching the academic rigor and cultural accuracy of the dataset.

[0098] S4.3. Based on the S4.2 pipeline, build task-oriented, large-scale data.

[0099] Based on the above pipeline, the system has carried out large-scale targeted data construction for five core tasks, and its key parameters and implementation methods are shown in Table 1.

[0100] Table 1 Key Parameters and Implementation Methods for the Five Core Tasks

[0101]

[0102] This phase utilizes an innovative data architecture and a hybrid annotation pipeline to systematically transform the deeply cleaned raw corpus into a high-quality, large-scale, multi-task instruction fine-tuning dataset that can be directly used for model training, providing a core driving force for shaping subsequent model capabilities.

[0103] S5. Quality Verification and Dataset Consolidation Phase: This phase verifies and consolidates the quality of the instruction-based fine-tuning dataset output from S4. This step may include the following sub-steps:

[0104] S5.1. Introduce a multi-dimensional quality sampling evaluation system to sample and evaluate the instruction fine-tuning dataset, and feed the results back to drive upstream optimization.

[0105] To provide quantifiable assurance of the quality of the final output dataset, this stage introduces a systematic, multi-dimensional quality sampling evaluation system. This system randomly samples the final data to be packaged according to a preset proportion (e.g., 1%), and then submits it to domain experts for manual review based on clearly defined evaluation criteria. Evaluation dimensions include, but are not limited to:

[0106] a. Content accuracy: Whether the output content is factually correct and logically coherent.

[0107] b. Format standardization: Whether the data architecture is fine-tuned in strict accordance with predefined instructions.

[0108] c. Privacy compliance: Has complete desensitization of sensitive information been completed, and is there any risk of information leakage?

[0109] d. Labeling rationality: Are the instructions clear, and do the input / output pairs match the task intent?

[0110] The core of this evaluation system lies in its closed-loop feedback mechanism: the evaluation results are quantified into specific indicators and transmitted back to the upstream links such as data cleaning (S3) and labeling (S4) to drive the iterative optimization of their processing strategies and models, thereby forming a self-improving quality control closed loop.

[0111] S5.2. Perform a standardized serialization and storage scheme to perform standardized serialization and persistent storage on the instruction fine-tuning dataset.

[0112] After quality verification, the system performs standardized serialization and persistent storage of the dataset. All data is uniformly encapsulated in JSONL format, with each line representing an independent JSON object. This format combines human readability with efficient machine processing capabilities, and due to its excellent streaming readability, it is natively supported by the vast majority of mainstream training frameworks, ensuring the dataset's universality and ease of use at the application level.

[0113] S5.3. A randomized partitioning strategy for the dataset is adopted to perform deterministic partitioning of the instruction fine-tuning dataset.

[0114] To facilitate model training and scientific evaluation, the system employs a randomization strategy to deterministically partition the complete dataset. Specifically, the partitioning is as follows:

[0115] (1) Training set: accounting for about 80% of the total data, serving as the main data source for training model parameters.

[0116] (2) Test set: accounting for about 20% of the total data, it is strictly isolated throughout the entire model training cycle and is only used to evaluate the generalization ability and performance on unknown data in the final evaluation, so as to ensure the objectivity and unbiasedness of the evaluation results.

[0117] This partitioning strategy, while ensuring data randomness, fixes the test set samples, providing a stable and consistent evaluation benchmark for performance comparison of different models or different training stages.

[0118] S6. Data Asset Management and Continuous Evolution Phase: This phase is used for the security management, version control, and continuous iteration of data assets.

[0119] S6.1. Deploy a secure storage and controlled access gateway:

[0120] To ensure the security and compliance of core data assets, the final dataset was deployed in an environment with stringent security measures. This step specifically included: employing a privately deployed storage solution (such as a private S3 or NAS) and integrating a fine-grained role-based access control (RBAC) model. This access control model isolates permissions and logs audits for access, read, and export operations of the dataset, ensuring that all data flows are traceable, thereby eliminating the risk of unauthorized access and data breaches at the infrastructure level.

[0121] S6.2. Implement version management and a full lifecycle traceability system to ensure data traceability and version reproducibility.

[0122] To address the continuous evolution of the dataset throughout its lifecycle, this invention implements a rigorous versioning management system. Each update, expansion, or correction of the dataset is assigned a unique, semantic version number (e.g., v1.0 baseline version, v1.1 cleaned and upgraded version, v2.0 multilingual extended version), supplemented by a structured change log. This log records in detail the differences between versions, the reasons for updates, and the changes in the data scale involved, thus constructing a complete and auditable traceability chain for the dataset from its creation to its final form, ensuring the reproducibility of research and practice.

[0123] S6.3. Implement a data-model co-evolution mechanism to drive continuous co-optimization of dataset and model capabilities.

[0124] The core feature of this system lies in its built-in data-model co-evolution mechanism. The system is not static, but rather operates as a dynamic, continuously optimized data factory. Based on performance evaluation feedback from the downstream model training phase, as well as the evolution of application scenarios and algorithmic technologies, it periodically and automatically re-triggers the complete data construction pipeline from task specification (S1) to quality verification (S5).

[0125] This mechanism drives incremental expansion of the dataset (such as adding new languages ​​or task types) and course-based scaling (such as increasing output length and complexity), thus creating a positive feedback loop between the dataset version and the model's capabilities. This closed loop of "evaluation-feedback-iteration-regeneration" ensures that the dataset and the models it serves can continuously adapt to new technological challenges and application needs, maintaining their cutting-edge nature and high availability.

[0126] Although the invention has been described with respect to a limited number of embodiments, those skilled in the art will understand from the foregoing description that other embodiments are conceivable within the scope of the invention described herein. Furthermore, it should be noted that the language used in this specification has been chosen primarily for readability and instructional purposes, and not for the purpose of explaining or limiting the subject matter of the invention.

Claims

1. A method for improving and constructing quality of low-resource, multilingual supervised datasets, characterized in that, include: S1, the task specification and multi-source acquisition phase, is used to define the task system and collect raw data; S2, the automated data acquisition and multimodal document parsing stage, is used to automatically acquire raw data and parse it into standardized text; S3, the semantic cleaning and privacy protection preprocessing stage, is used for deep cleaning of text, segmentation of semantic blocks and desensitization of privacy information; S4, the structured encapsulation and human-machine collaborative annotation stage, is used to transform the cleaned text into high-quality instruction fine-tuning data. S4 includes: S4.1, designing a unified instruction fine-tuning data architecture to map diverse task types into machine-readable structured formats; S4.2, building a human-machine collaborative hybrid annotation pipeline based on the data architecture designed in S4.1; and S4.3, building task-oriented large-scale data based on the pipeline in S4.

2. S5, the quality verification and dataset solidification stage, is used to perform quality verification and solidification on the instruction fine-tuning dataset output by S4. S6, the data asset management and continuous evolution stage, is used for the security management, version control and continuous iteration of data assets; S1 includes: S1.1, Multi-task text system definition and specification, which clearly defines five core task types and their competency development objectives: a. Open-ended question answering: This aims to train the model's ability to generate long texts and detailed factual content for open-domain questions; b. Long text summarization: By requiring the model to condense information from long documents, this aims to challenge and improve its deep understanding and core information extraction capabilities. c. Multi-turn dialogue: Simulates real user interaction scenarios with AI, covering daily communication, task-oriented dialogue and role-playing, aiming to enhance the model's ability to maintain context and follow instructions; d. Text continuation: Based on the sequence generation nature of LLMs, three structured subclasses were designed: continuation, imitation, and style transfer, aiming to systematically evaluate and improve the model's ability to generate long texts coherently and adapt to different styles. e. Heuristic text generation: The model is required to generate complete and original text content based on open-ended prompts, aiming to stimulate its imagination, context adaptability and generation diversity; S1.2, Implement multi-source heterogeneous data acquisition; S1.3, Embedded Security and Privacy Compliance Framework: This framework has formulated clear rules for identifying and de-identifying sensitive information based on the characteristics of text in different languages. Through regular expressions and named entity recognition technology, it automatically locates and replaces personal identity information. At the same time, it implements strict access control policies, and through identity authentication and access logs, it ensures that all data entering the dataset does not contain any unauthorized content, thus guaranteeing the legality and compliance of data use from the source.

2. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 1, characterized in that, S2 include: S2.1, Build an automated data collection and metadata management engine, which is used to achieve large-scale and permission-based acquisition of raw data; S2.2, Multimodal Document Parsing and Intermediate Representation Generation: Design a unified multimodal document parsing and conversion pipeline. For mainstream document formats, integrate and schedule a dedicated parsing toolchain to uniformly convert heterogeneous documents into standardized plain text format, forming the intermediate representation required for downstream processing.

3. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 2, characterized in that, In S2.1, during the collection process, the automated collection and metadata management engine automatically extracts and persistently stores key metadata related to document source and quality, including: document creation date, collector identity, document type, and original Uniform Resource Locator.

4. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 1, characterized in that, S3 include: S3.1 deploys a multi-level noise cleanup and structured information extraction engine for deep cleaning and structure enhancement of standardized text; S3.2, Execute the context-aware semantic segmentation algorithm to segment the text; S3.3, Construct a text modality privacy-sensitive information desensitization system, thereby using standardized placeholders to replace the original sensitive information; S3.4, Redundancy control and data reduction processing.

5. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 4, characterized in that, S3.1 includes: The multi-level noise removal and structured information extraction engine automatically identifies and removes irrelevant text elements that interfere with semantic understanding through pattern matching and rule bases, including headers and footers, page numbers, legal disclaimers, and duplicate content; at the same time, the engine performs character-level normalization to clean up redundant line breaks, broken spaces, and UTF-8 incorrect encoding. For non-plain text elements in the document, the engine integrates a dedicated processing module: Table structuring conversion module: Extracts and converts the identified table data into Markdown structured text format to preserve its row and column relationships and data semantics; Image Optical Character Recognition (OCR) module: Performs evaluative extraction of text content embedded in images.

6. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 4, characterized in that, In S3.2, the context-aware semantic segmentation algorithm uses a recursive character text segmenter: it actively identifies semantic integrity boundaries in the text and prioritizes segmentation on such boundaries; to prevent semantic breaks in the context caused by segmentation, the algorithm introduces a controllable overlap area between adjacent text blocks, thereby maintaining the coherence and integrity of the long document context during model training.

7. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 4, characterized in that, In S3.3, a two-layer recognition architecture is adopted: Pattern matching layer: Based on predefined regular expressions, it provides high-precision matching of sensitive information with fixed patterns; Entity recognition layer: Utilizes a pre-trained natural language processing model to identify and locate personal names, place names, and organization names in unstructured text; All identified sensitive information is systematically replaced with uniform, meaningless, standardized placeholders.

8. The method for improving and constructing quality of low-resource multilingual supervised datasets according to claim 1, characterized in that, S5 include: S5.1 introduces a multi-dimensional quality sampling evaluation system to sample and evaluate the instruction fine-tuning dataset and feed the results back to drive upstream optimization; S5.2, executes a standardized serialization and storage scheme to perform standardized serialization and persistent storage on the instruction fine-tuning dataset; S5.3 employs a randomized partitioning strategy for the dataset to perform deterministic partitioning of the instruction fine-tuning dataset.

Images