Model training-oriented data set construction method and device

Abstract

The invention discloses a model training-oriented data set construction method and device, and relates to the technical field of data processing. The method comprises the following steps: constructing a data carding agent based on a large language model, receiving multi-source original data and target domain business requirements through the agent, and screening out a high-association effective data set by utilizing deep semantic analysis and vectorization matching; the data set is input into an automatic core processing assembly line, multi-modal data standardization, high-quality question and answer pair construction and safety quality enhancement processing are executed in sequence, and a standard data set for target domain large model training is generated; performing model training and evaluation based on the data set, and collecting performance feedback data of the model on a business task to construct a closed loop iterative optimization mechanism; and according to the performance feedback, reversely positioning data production link defects, and dynamically adjusting a screening standard or a processing rule until the model performance meets a preset requirement. According to the method, the problems of data semantic deficiency and difficult quality iteration are effectively solved.

Description

Technical Field

[0001] This invention relates to the field of data processing technology, and in particular to a method and apparatus for constructing datasets for model training. Background Technology

[0002] With the deepening of digital transformation, various vertical industries (such as smart cities, industrial manufacturing, and financial risk control) have accumulated massive amounts of operational data. This data exhibits significant multi-source heterogeneity: it includes structured database records from business systems, continuous time-series data from IoT devices, and a large amount of unstructured normative documents, operation manuals, images, and video data.

[0003] Currently, data processing for model training primarily relies on traditional "pipeline cleaning" or "data lake" aggregation architectures. Existing technologies typically employ Extract-Transform-Load (ETL) tools to physically aggregate scattered data according to predefined rules, loading it into a data warehouse through deduplication, regular expression filtering, and simple format conversion. The core objective of this processing model is often to meet the needs of Business Intelligence (BI) reporting and statistical tasks or traditional supervised learning tasks.

[0004] However, existing technologies have revealed significant limitations in meeting the current demands of cognitive intelligence applications based on large language models. First, current technologies focus on the "physical stacking" of data rather than "semantic filtering." During the data aggregation phase, the lack of analytical capabilities to understand the deep semantic relationships between data content and the business needs of the target domain leads to a large amount of low-value or irrelevant data being mixed into the training set, diluting the density of key knowledge. Second, the data produced by existing processes is often fragmented "records," lacking the coherence and causal logic required for training large models. Field relationships in business databases are often implicit in table structure definitions, and when converted to text, the business logic context is easily lost, making it difficult for large models to learn the evolutionary patterns of events and prone to illusions during reasoning. Finally, existing data processing workflows are mostly unidirectional, open-loop "pipeline" operations. Once cleaning rules and filtering criteria are set, it is difficult to dynamically adjust them based on the actual performance of downstream models. When the target domain's large model performs poorly on a specific task, there is a lack of automated feedback mechanisms to diagnose problems in the upstream data links, requiring maintenance personnel to spend a significant amount of time on manual troubleshooting, severely hindering the effective implementation of large industry models. Summary of the Invention

[0005] This invention provides a method and apparatus for constructing datasets for model training, solving the problems of coarse data selection, loss of business logic semantics, and lack of closed-loop optimization mechanisms in existing technologies. To achieve the above objectives, the embodiments of this invention adopt the following technical solutions: In a first aspect, embodiments of the present invention propose a dataset construction method for model training, the method comprising: Construct a data sorting agent based on a large language model. The data sorting agent receives raw data from multiple sources and business requirements of the target domain. It performs deep semantic analysis and vector matching on the raw data from multiple sources and filters out highly relevant and effective data sets. The highly correlated and effective dataset is input into the automated core processing pipeline, where multimodal data standardization, high-quality question-answer pair construction, and security enhancement are performed sequentially to generate a standard target domain dataset for training large models in the target domain. Training and evaluation of large target domain models are conducted based on standard target domain datasets. Performance feedback data of large target domain models on business tasks are collected, and a closed-loop iterative optimization mechanism is constructed. Based on performance feedback data, defects in the data production process are identified through reverse analysis, and the screening criteria of the data processing agent or the processing rules of the core processing pipeline are dynamically adjusted until the performance of the target domain model meets the preset requirements.

[0006] Preferably, a data processing intelligent agent based on a large language model is constructed. This agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out highly relevant and effective data sets, including: Pre-determine the specific capability requirements of the target domain business, and use the specific capability requirements as the target semantic benchmark; The big language model is used to extract the meta-information and content features of multi-source raw data, and the target semantic benchmark and content features are mapped into demand vectors and data vectors in high-dimensional semantic space, respectively. Calculate the cosine similarity between the demand vector and the data vector to obtain a quantitative relevance score; Determine whether the relevance score exceeds a preset decision threshold. If it does, the corresponding data is determined to be highly relevant and valid data and included in the set of highly relevant and valid data. Otherwise, the corresponding data is marked as low-relevance data and isolated.

[0007] Preferably, the highly correlated and effective data set is input into the automated core processing pipeline, and multimodal data standardization processing is performed sequentially, including: For unstructured document data, deep parsing tools are used to extract text, tables, and page layout information, and then convert them into standardized text. For table structure data in the business database, execute business logic while maintaining the transformation; Business logic preservation transformation includes: parsing the field definitions and inter-table relationships of the data table, and reconstructing the discrete field information into natural language description paragraphs that retain the logical dependencies between fields and the business context according to preset business rules; For semi-structured data from portal websites, we parse the webpage structure and sort out the information relationships, converting the semi-structured data from portal websites into text corpus in a unified format.

[0008] Preferably, the business logic conversion further includes: Identify key business entities and process nodes in the data table; Analyze the preconditions and subsequent flow relationships between process nodes; The process nodes and flow relationships are transformed into narrative text containing causal logic to preserve the semantics of the business chain in the original data and prevent the loss of business logic context during the textification process.

[0009] Preferably, the construction of high-quality question-answer pairs includes: The design covers a standardized question-and-answer framework that includes multiple choice, fill-in-the-blank, and essay questions. Based on the standardized text corpus obtained after multimodal data standardization processing, the business knowledge of domain experts is used for manual annotation to generate initial question-answer pairs that fit the actual business scenario. A cross-validation mechanism is introduced, in which domain experts from different groups review and correct the accuracy and logic of the initial question-answer pairs, generating high-quality question-answer pair data.

[0010] Preferably, the safety and quality enhancement treatment includes: Sensitive information identification algorithms are used to detect sensitive data in standardized text corpora and high-quality question-answer pairs, and then anonymization processing is performed. Perform refined data cleaning to remove invalid characters, advertising links, and redundant duplicate data; The cleaned data is packaged and stored in a specific format required for training large models in the target domain.

[0011] Preferably, a large-scale target domain model is trained and evaluated based on a standard target domain dataset, performance feedback data of the large-scale target domain model on business tasks is collected, and a closed-loop iterative optimization mechanism is constructed, including: During the model training or testing phase, monitor the output results of the large model in the target domain on specific normative document understanding or business process reasoning tasks; Analyze the error types in the output results. Error types include factual errors and logical reasoning errors. The error type and corresponding error sample are fed back to the data production stage to trigger the corresponding optimization strategy.

[0012] Preferably, based on performance feedback data, defects in the data production process are identified through reverse analysis, and the screening criteria of the data processing agent or the processing rules of the core processing pipeline are dynamically adjusted, including: When frequent factual errors are detected, it is determined to be a defect in the quality of basic data, and the first optimization instruction is generated to correct the data cleaning rules or standardization transformation rules in the core processing pipeline. When frequent logical reasoning errors are detected or the large model of the target domain cannot cope with the subdivided scenarios, it is determined that the data logic chain or scenario coverage is insufficient. A second optimization instruction is generated to adjust the annotation strategy of high-quality question-answer pairs or optimize the decision threshold of the data sorting agent.

[0013] Preferably, the method further includes: During the closed-loop iterative optimization process, the dataset version and corresponding model performance metrics for each round are recorded. Establish a mapping relationship between data versions and the capabilities of large-scale models in the target domain. By comparing and analyzing the impact of different dataset versions on the performance of large-scale models in the target domain, continuously update the pre-set knowledge base of the data processing agent.

[0014] Secondly, embodiments of the present invention propose a dataset construction apparatus for model training, the apparatus comprising: The data sorting agent module is used to build a data sorting agent based on a large language model. The data sorting agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out highly relevant and effective data sets. The core pipeline processing module is used to input highly correlated and effective datasets into the automated core processing pipeline, which sequentially performs multimodal data standardization processing, high-quality question-answer pair construction, and security quality enhancement processing to generate a standard target domain dataset for training large models in the target domain. The closed-loop iterative optimization module is used to train and evaluate large target domain models based on standard target domain datasets, collect performance feedback data of large target domain models on business tasks, and construct a closed-loop iterative optimization mechanism. Based on the performance feedback data, it reverse-locates the defects in the data production process and dynamically adjusts the screening criteria of the data sorting agent module or the processing rules of the core pipeline processing module until the performance of the large target domain model meets the preset requirements.

[0015] Beneficial effects: This invention provides a dataset construction method and apparatus for model training. By constructing a data processing agent based on a large language model, it performs deep semantic analysis and vectorized matching on multi-source raw data. This accurately identifies and filters effective data highly relevant to the business needs of the target domain, effectively solving the problems of low relevance and high noise caused by traditional rule-based filtering, and improving the knowledge density of training data. Through an automated core processing pipeline, multimodal data standardization, high-quality question-answer pair construction, and security quality enhancement are executed sequentially, transforming heterogeneous data into a unified format while preserving the logical dependencies and causal relationships between data. This provides a standard training dataset rich in contextual logic for large models in the target domain, enhancing the model's understanding and reasoning capabilities in complex business scenarios. It changes the traditional one-way data processing mode, enabling real-time collection of model performance feedback on business tasks, reverse location of defects in the data production process, and dynamic adjustment of filtering criteria or processing rules. This mechanism significantly reduces manual maintenance costs, ensures the continuous output of high-quality data that adapts to the model's shortcomings, and guarantees the stability and continuous optimization of the performance of large models in the target domain. Attached Figure Description

[0016] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the invention. Furthermore, the same reference numerals denote the same parts throughout the drawings. In the drawings: Figure 1 This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention; Figure 2 This is a flowchart illustrating the steps of a dataset construction method for model training provided in an embodiment of the present invention. Figure 3 This is a schematic diagram of the functional modules of a dataset construction device for model training provided in an embodiment of the present invention. Detailed Implementation

[0017] To more clearly illustrate the technical solutions in the embodiments of the invention or the prior art, the invention will be briefly introduced below in conjunction with the accompanying drawings and descriptions of the embodiments or the prior art. Obviously, the following description of the structure of the drawings is merely some embodiments of the invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort. It should be noted that the description of these embodiments is for the purpose of aiding understanding the invention, but does not constitute a limitation on the invention. The solution of the present invention will be further described below in conjunction with the accompanying drawings.

[0018] Reference Figure 1 , Figure 1This is a schematic diagram of the electronic device structure of the hardware operating environment involved in the embodiments of the present invention.

[0019] like Figure 1 As shown, the electronic device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0020] Those skilled in the art will understand that Figure 1 The structure shown does not constitute a limitation on the electronic device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0021] like Figure 1 As shown, the memory 1005, serving as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and electronic programs. In the illustrated electronic device, the network interface 1004 is primarily used for data communication with a network server; the user interface 1003 is primarily used for data interaction with the user; the processor 1001 and memory 1005 in the electronic device of this invention can be disposed within the electronic device. The electronic device uses the processor 1001 to call the dataset construction device for model training stored in the memory 1005 and executes the dataset construction method for model training provided in this embodiment of the invention.

[0022] Reference Figure 2 This invention provides a dataset construction method for model training. This method aims to address the problem of how to transform fragmented, multi-source, heterogeneous raw data into high-quality, semantically rich large-scale model training corpora in target business domains (such as complex business domains or large enterprise management domains), and how to continuously optimize data quality through a closed-loop feedback mechanism. The method includes the following steps S101 to S104: S101. Construct a data sorting intelligent agent based on a large language model. The data sorting intelligent agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out highly relevant and effective data sets.

[0023] In existing technologies, data aggregation often employs simple keyword matching or regular expression filtering, which struggles to understand the deep semantics of the data. For example, in a target business domain, a document titled "Several Measures to Improve Business Operation Efficiency" may not directly contain the keyword "project startup speed," yet its content is highly relevant to the business requirements. This step introduces a data processing agent based on a Large Language Model (LLM), leveraging its powerful semantic understanding capabilities to achieve a leap from "formal matching" to "semantic matching." Specifically, step S101 includes the following sub-steps: S1011. Preset the specific capability requirements of the target domain business and use the specific capability requirements as the target semantic benchmark.

[0024] In the initialization phase, the first step is to define the specific capability requirements of the target domain's business. These requirements typically originate from the experience of business experts or guidance documents from higher authorities. For example, in a business service scenario, specific capability requirements might include "optimizing business processing procedures," "improving asset registration efficiency," "strengthening compliance supervision," and "implementing resource introduction strategies." Subsequently, the text describing these requirements in natural language is used as "anchor points" or "target semantic benchmarks" for subsequent data filtering. These benchmarks are not just phrases but can also be paragraphs containing detailed descriptions. For example, "optimizing business processing procedures" can be further expanded to "shortening business registration time, simplifying business approval processes, promoting one-stop online services, and lowering market entry barriers." The more specific the description, the more accurate the semantic benchmark.

[0025] S1012. Utilize a large language model to extract metadata and content features from multi-source raw data, and map the target semantic benchmark and content features into demand vectors and data vectors in a high-dimensional semantic space, respectively.

[0026] This step involves accessing and extracting features from multiple sources of raw data. The raw data includes unstructured data such as normative guidance documents (PDF / Word), meeting minutes, and research reports; structured data such as item databases and processing record tables in business approval systems; and semi-structured data such as business guide pages and Q&A records in interactive communication sections of official service portals. For each piece of raw data accessed, the data processing agent first extracts metadata such as the data's title, issuing organization, publication time, and tags. For long texts (such as normative documents dozens of pages long), direct vectorization may be limited by the model's context window length or introduce too much noise. Therefore, a large language model is first used to generate a summary or core feature description of the content. For example, given the input "Notice on Issuing the Action Plan for Digital Transformation of Small and Medium-sized Enterprises," the model outputs the summary: "This document mainly involves normative documents supporting the financial support for the digital transformation of small and medium-sized enterprises, the construction of technical service platforms, and personnel training plans." Finally, using pre-trained text embedding models, such as BERT, RoBERTa, or embedding models fine-tuned for Chinese vertical domains, the "target semantic benchmark" and the data's "content features" (meta-information + summary) are mapped to the same high-dimensional semantic space. Let the target semantic benchmark be... The characteristics of the data content are The mapping function is The generated demand vector is The generated data vector is Vector dimensions are typically 768, 1024 or higher, with each dimension representing a potential semantic feature.

[0027] S1013. Calculate the cosine similarity between the demand vector and the data vector to obtain a quantitative correlation score.

[0028] After obtaining the demand vector and data vector Next, it is necessary to quantify the semantic correlation between the two. This embodiment uses cosine similarity as the metric. Cosine similarity evaluates the similarity between two vectors by calculating the cosine of the angle between them. The closer the value is to 1, the more similar they are; the closer it is to 0, the less related they are; and the closer it is to -1, the opposite their semantics.

[0029] Correlation score The calculation formula is as follows: in, Represents the dot product of vectors. and These represent the magnitude of the vector (EuclideanNorm). This represents the dimension of the vector space. and They represent the vector at the th order. The formula calculates the relevance score for even texts that are completely different in meaning (such as "business environment optimization" and "reducing the number of business processing steps") because they are close in the high-dimensional semantic space.

[0030] S1014. Determine whether the correlation score exceeds the preset decision threshold. If it does, determine that the corresponding data is highly correlated and valid data and include it in the set of highly correlated and valid data. Otherwise, mark the corresponding data as low-correlation data and isolate it.

[0031] Preset a decision threshold (For example This threshold is not fixed and can be dynamically adjusted based on feedback from subsequent step S104. During the screening process, if the relevance score... If the data is highly relevant to business needs, it is marked as "highly relevant and valid data" and stored in the "highly relevant and valid data set" pending processing area; if If the data is deemed noise or low-value (such as simple meeting notifications or briefings without substantial content), it is marked as "low-relevance data." Low-relevance data is not physically deleted directly; instead, it is moved to a "cold storage area" or "isolation area" to prevent accidental deletion and ensure that this data can still be reactivated and scanned when subsequent business needs change or thresholds are adjusted.

[0032] S102. Input the highly correlated and effective data set into the automated core processing pipeline, and sequentially perform multimodal data standardization processing, high-quality question-answer pair construction and security quality enhancement processing to generate a standard target domain dataset for training large models in the target domain.

[0033] This step is the "processing plant" for data processing, designed to transform the selected raw materials (highly correlated data) into refined feed (standard datasets) that can be directly "consumed" by large models. This pipeline emphasizes automation, standardization, and logical integrity. Specifically, step S102 includes the following sub-steps: S1021. For unstructured document data, use deep parsing tools to extract text, table and layout structure information, and convert it into standardized text.

[0034] Unstructured documents (such as PDF format specification documents) often contain complex layout elements, such as multi-page tables, two-column layouts, headers and footers, which traditional text extraction tools often mix together. Therefore, this step first uses a deep learning-based layout analysis model (such as LayoutLMv3) to extract visual features from the document image, identifying different areas such as titles, paragraphs, tables, image captions, and page numbers. Next, for scanned PDFs or images, an OCR engine (such as PaddleOCR) is used to recognize the text content, and combined with the layout analysis results, the text flow is reconstructed according to human reading order. Simultaneously, for tables in the document, not only is the text within cells recognized, but also the row and column structure of the table is identified, converting it into Markdown or HTML table code and embedding it into the body text. Finally, the document is converted to a unified Markdown or JSONL format, preserving the heading hierarchy and making the structured features of the data explicit.

[0035] S1022. For table structure data in the business database, execute business logic to maintain the transformation.

[0036] Traditional database export is typically in CSV format, which is a simple row and column layout. This method loses the foreign key relationships between tables, the business meaning of fields, and the hierarchical logic of the data. This step performs "business logic preservation transformation," first reading the database's data dictionary to obtain metadata such as table names, field names, field comments, primary keys, and foreign key constraints, and then constructing an entity relationship diagram based on the foreign key relationships. The specific execution logic is as follows: S1022a to S1022c: S1022a. Identify key business entities and process nodes in the data tables. For example, in a business approval system, identify the main table t_approval_item (business item), whose key fields include item_name (item name) and handling_dept (handling agency). Also identify the associated tables t_material (business materials) and t_process (business process nodes). Business entities are business items, and process nodes include stages such as acceptance, review, approval, and completion.

[0037] S1022b, Analyze the preconditions and subsequent flow relationships between process nodes.

[0038] By analyzing the pre_node_id (previous node ID) and next_node_id (next node ID) in the t_process table, the workflow logic can be determined. For example, the analysis shows that the "on-site verification" node can only proceed after the "initial review of materials" node has passed.

[0039] S1022c: Transform process nodes and flow relationships into narrative text containing causal logic to preserve the semantics of the business chain in the original data and prevent the loss of business logic context during the textification process.

[0040] Using a pre-defined template engine or generative model, the structured data described above is transformed into natural language paragraphs. For example, a table record containing discrete data such as "Business Operation License," "20-day time limit," "Acceptance," and "Verification" can be transformed into "Regarding the application for a 'Business Operation License,' the stipulated processing time limit is 20 days. The application process is as follows: First, the first step, 'Acceptance,' is conducted; after acceptance, the second step, 'Verification,' is initiated. These two steps have a strict sequential dependency." Through this transformation, the "hard logic" in the database is converted into "soft corpus" that the large model can understand.

[0041] S1023. For the semi-structured data of portal websites, parse the webpage structure and sort out the information relationships, and convert the semi-structured data of portal websites into text corpus in a unified format.

[0042] This step first parses the HTML Document Object Model (DOM) to remove noisy elements such as navigation bars, advertisements, and copyright notices. Then, it organizes and correlates information, identifying links such as "Related Reading" and "Attachment Download" on the webpage, crawling related page content, and merging them into a complete document chunk. For example, the content of the "Business Operation Guide" webpage and its attachment "Application Form Template" is merged to form a complete information unit.

[0043] S1024. High-Quality Question-Answer Pair Construction: To support instruction tuning of the large model, the standardized corpus needs to be transformed into question-answer pairs (QA pairs). First, a standardized question-answering framework covering multiple question types, including single-choice, multiple-answer, fill-in-the-blank, and essay questions, is designed. For example, single-choice / multiple-answer questions assess the model's memory of exact facts, fill-in-the-blank questions assess the model's ability to extract key entities or data, and essay questions assess the model's logical reasoning and summarizing abilities. Next, based on the standardized text corpus obtained after multimodal data standardization, an initial QA pair tailored to actual business scenarios is generated using a "human-machine collaboration" model: first, a general strong model automatically generates candidate QA pairs based on the corpus; then, domain experts (such as senior business personnel in business processing centers) verify and refine these pairs, injecting "tacit knowledge" from actual business operations (such as practical experience in handling incomplete applications). Finally, a cross-validation mechanism is introduced to implement "double-blind validation," in which domain experts from different groups review and correct the accuracy, security, and logic of the initial question-and-answer pairs. Only data that passes the validation can be included in the final dataset.

[0044] S1025. Enhanced security and quality processing: Before data enters the model for training, strict security and quality control must be carried out.

[0045] First, sensitive information identification algorithms (such as NER technology) are used to detect sensitive data in standardized text corpora and high-quality question-answer pairs, identifying personal privacy information (PII) such as ID card numbers, mobile phone numbers, and home addresses, as well as sensitive business data, and performing de-identification processing such as masking or replacement. Second, refined data cleaning is performed, using MinHash or SimHash algorithms to remove document deduplication, preventing the model from remembering repeated information and causing overfitting, and removing residual HTML tags and garbled characters. Finally, the cleaned data is packaged and stored according to the specific format (such as JSONL) required for training large-scale models in the target domain, adapting it to mainstream large-scale model training frameworks.

[0046] S103. Train and evaluate the target domain large model based on the standard target domain dataset, collect performance feedback data of the target domain large model on business tasks, and build a closed-loop iterative optimization mechanism.

[0047] This step no longer treats the dataset as a static output, but rather places it within the dynamic lifecycle of model training. Specifically, step S103 includes the following sub-steps: S1031. During the model training or testing phase, monitor the output results of the target domain large model on specific normative document understanding or business process reasoning tasks.

[0048] During the training phase, the focus is on observing the training loss curve and monitoring the model's perplexity metric on the validation set. In the testing and evaluation phase, a dedicated "business capability test set" (Benchmarks) is constructed, allowing the model to answer a series of complex business questions. The evaluation method combines manual and automated assessment; that is, automated scoring scripts are used to calculate BLEU and ROUGE scores, while human experts are also involved in scoring.

[0049] S1032. Analyze the error types in the output results. Error types include factual errors and logical reasoning errors.

[0050] Deep attribution analysis was conducted on samples where the model gave incorrect answers. The first category is factual errors, where the model fabricated non-existent business rules or terms, or misrepresented data values ​​(e.g., stating "5 million" instead of "500,000" in funding support). This usually indicates noise in the training data or incomplete data cleaning, leading the model to learn incorrect information. The second category is logical reasoning errors, where the model knows all the facts (e.g., knows condition A and consequence B), but cannot deduce the correct conclusion (e.g., cannot determine whether the user meets condition A), or contradicts itself in multiple rounds of dialogue. This usually indicates low quality of the question-and-answer pairs, a lack of Chain of Thought (CoT) guidance, or that key logical connections were missed during the data processing phase.

[0051] S1033. Feed back the error type and corresponding error sample to the data production stage to trigger the corresponding optimization strategy.

[0052] A "negative sample library" or "incorrect question set" will be established, and these incorrect cases will be fed back to the data processing control center as a signal to trigger S104 optimization.

[0053] S104. Based on performance feedback data, reverse-locate the defects in the data production process, and dynamically adjust the screening criteria of the data sorting agent or the processing rules of the core processing pipeline until the performance of the target domain large model meets the preset requirements.

[0054] This step completes the data engineering loop (Data-Centric AI). It involves not only tuning model parameters, but also tuning the data itself. Specifically, step S104 includes the following sub-steps: S1041. When factual errors are detected to occur frequently, it is determined to be a defect in the quality of basic data. The first optimization instruction is generated to correct the data cleaning rules or standardization transformation rules in the core processing pipeline.

[0055] When the model frequently makes factual errors, such as answering "2020 old rules" as "2023 rules," it is determined to be a defect in the basic data quality. The cause may be that "expired" files were not removed during the data cleaning stage. In response, a "first optimization instruction" is automatically generated to update the cleaning rules in S1025, adding "timeliness detection" logic, or during the S1021 standardization stage, the "effective date" and "expiration date" fields of the files are forcibly extracted, and outdated data is marked. <expired>Label or remove directly.

[0056] S1042. When frequent logical reasoning errors are detected or the large model of the target domain cannot cope with the subdivided scenarios, it is determined that the data logic chain or scenario coverage is insufficient. A second optimization instruction is generated to adjust the annotation strategy of high-quality question-answer pairs or optimize the decision threshold of the data sorting agent.

[0057] When frequent logical reasoning errors are detected, such as the model's inability to handle complex "combined business inquiries" (e.g., whether a business can be processed if conditions A, B, and C are met simultaneously), it is determined that the data logic chain or scenario coverage is insufficient. This may be because the question-answer pairs constructed in S1024 are too simple, lacking multi-hop reasoning samples, or because S101 missed some highly explanatory detailed documents during data screening. In response, a "second optimization instruction" is generated. On one hand, the annotation strategy of S1024 is adjusted, requiring experts to add "thinking chain" type question-answer pairs and explicitly write out the reasoning steps; on the other hand, the decision threshold of S1014 is adjusted, appropriately lowering the decision threshold. (For example, reducing it from 0.75 to 0.70), introducing more edge-related explanatory documentation to increase knowledge coverage.

[0058] S1043. During the closed-loop iterative optimization process, record the dataset version and corresponding model performance indicators for each round; establish a mapping relationship between the data version and the capabilities of the target domain large model; and continuously update the preset knowledge base of the data processing agent by comparing and analyzing the impact of different dataset versions on the performance of the target domain large model.

[0059] Throughout the optimization process, data version control tools (such as DVC) are used to record the dataset version (e.g., Dataset V1.0, V1.1...) and corresponding model performance metrics for each round. Simultaneously, a mapping relationship is established between data versions and the capabilities of large models in the target domain. Through comparative analysis, it is identified which data features contribute most to improving model performance. These data features, proven to be "high-value" (such as specific keyword combinations or document structures), are then written back into the prompts or knowledge base of the S101 data processing agent, making it more intelligent and accurate in the next round of new data selection.

[0060] Through the iterative process from S101 to S104, the final standard target domain dataset not only meets the training requirements of large models in terms of scale, but also achieves high standards in terms of quality, logic, and security. It can significantly improve the performance of large target domain models in actual business and solve the problem of the difficulty in deploying large models in vertical domains.

[0061] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

[0062] This invention also provides a dataset construction apparatus for model training, referring to... Figure 3 The diagram illustrates a functional block diagram of a dataset construction device 300 for model training according to the present invention. The device may include the following modules: The data sorting agent module 301 is used to construct a data sorting agent based on a large language model. The data sorting agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out a highly relevant and effective data set. The core pipeline processing module 302 is used to input the highly correlated effective data set into the automated core processing pipeline, and sequentially perform multimodal data standardization processing, high-quality question-answer pair construction and security quality enhancement processing to generate a standard target domain dataset for training large models in the target domain. The closed-loop iterative optimization module 303 is used to train and evaluate the target domain large model based on the standard target domain dataset, collect the performance feedback data of the target domain large model on business tasks, and construct a closed-loop iterative optimization mechanism. The dynamic adjustment module 304 is used to reverse locate the defects in the data production process based on the performance feedback data, and dynamically adjust the screening criteria of the data sorting agent or the processing rules of the core processing pipeline until the performance of the target domain large model meets the preset requirements.

[0063] Based on the same inventive concept, another embodiment of the present invention provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus. Memory, used to store computer programs; The processor, when executing a program stored in memory, implements the dataset construction method for model training according to the present invention.

[0064] The communication bus mentioned above can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into address bus, data bus, control bus, etc. For ease of illustration, only one thick line is used in the diagram, but this does not indicate that there is only one bus or one type of bus. The communication interface is used for communication between the aforementioned terminal and other devices. The memory can include Random Access Memory (RAM), or non-volatile memory, such as at least one disk storage device. Optionally, the memory can also be at least one storage device located remotely from the aforementioned processor.

[0065] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0066] Furthermore, to achieve the above objectives, embodiments of the present invention also propose a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the dataset construction method for model training according to embodiments of the present invention.

[0067] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable hardware devices (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0068] The embodiments of the present invention are described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (apparatus), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0069] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0070] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0071] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. "And / or" indicates that either one or both can be chosen. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes the element.

[0072] The above are merely specific embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any person skilled in the art can easily conceive of various equivalent modifications or substitutions within the technical scope disclosed in the present invention, and these modifications or substitutions should all be covered within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.< / expired>

Claims

1. A method for constructing a dataset for model training, characterized in that, The method includes: A data sorting agent based on a large language model is constructed. The data sorting agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out highly relevant and effective data sets. The highly correlated and effective dataset is input into the automated core processing pipeline, where multimodal data standardization, high-quality question-answer pair construction, and security quality enhancement are performed sequentially to generate a standard target domain dataset for training large models in the target domain. Based on the standard target domain dataset, a large target domain model is trained and evaluated, and performance feedback data of the large target domain model on business tasks is collected to construct a closed-loop iterative optimization mechanism. Based on the performance feedback data, the defects in the data production process are located in reverse, and the screening criteria of the data sorting agent or the processing rules of the core processing pipeline are dynamically adjusted until the performance of the target domain large model meets the preset requirements.

2. The dataset construction method for model training according to claim 1, characterized in that, The aforementioned data processing intelligent agent, based on a large language model, receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out a highly relevant and effective data set, including: The specific capability requirements of the target domain business are preset, and the specific capability requirements are used as the target semantic benchmark; The large language model is used to extract the meta-information and content features of the multi-source raw data, and the target semantic benchmark and the content features are respectively mapped into demand vector and data vector in high-dimensional semantic space. Calculate the cosine similarity between the demand vector and the data vector to obtain a quantitative relevance score; Determine whether the correlation score exceeds a preset decision threshold. If it does, determine that the corresponding data is highly correlated and valid data and include it in the set of highly correlated and valid data. Otherwise, mark the corresponding data as low-correlation data and isolate it.

3. The dataset construction method for model training according to claim 1, characterized in that, The process of inputting the highly correlated and effective data set into the automated core processing pipeline and sequentially performing multimodal data standardization processing includes: For unstructured document data, deep parsing tools are used to extract text, tables, and page layout information, and then convert them into standardized text. For table structure data in the business database, execute business logic while maintaining the transformation; The business logic preservation transformation includes: parsing the field definitions and inter-table relationships of the data table, and reconstructing the discrete field information into natural language description paragraphs that retain the logical dependencies between fields and the business context according to preset business rules; For the semi-structured data of portal websites, the webpage structure is parsed and the information relationships are sorted out, and the semi-structured data of the portal websites is converted into text corpus in a unified format.

4. The dataset construction method for model training according to claim 3, characterized in that, The business logic conversion further includes: Identify key business entities and process nodes in the data table; Analyze the preconditions and subsequent flow relationships between the process nodes; The process nodes and flow relationships are transformed into narrative text containing causal logic to preserve the semantics of the business chain in the original data and prevent the loss of business logic context during the textification process.

5. The dataset construction method for model training according to claim 3, characterized in that, The construction of the high-quality question-answer pairs includes: The design covers a standardized question-and-answer framework that includes multiple choice, fill-in-the-blank, and essay questions. Based on the standardized text corpus obtained after the multimodal data standardization process, manual annotation is performed using the business knowledge of domain experts to generate initial question-and-answer pairs that fit the actual business scenario. A cross-validation mechanism is introduced, in which domain experts from different groups review and correct the accuracy and logic of the initial question-answer pairs to generate high-quality question-answer pair data.

6. The dataset construction method for model training according to claim 5, characterized in that, The safety and quality enhancement process includes: Sensitive information identification algorithms are used to detect sensitive data in the standardized text corpus and the high-quality question-answer pairs, and desensitization processing is performed. Perform refined data cleaning to remove invalid characters, advertising links, and redundant duplicate data; The cleaned data is packaged and stored in a specific format required for training the large model in the target domain.

7. The dataset construction method for model training according to claim 1, characterized in that, The process of training and evaluating a large target domain model based on the standard target domain dataset, collecting performance feedback data of the large target domain model on business tasks, and constructing a closed-loop iterative optimization mechanism includes: During the model training or testing phase, monitor the output results of the target domain large model on specific normative document understanding or business process reasoning tasks; Analyze the error types in the output results, which include factual errors and logical reasoning errors; The error types and corresponding error samples are fed back to the data production stage to trigger corresponding optimization strategies.

8. The dataset construction method for model training according to claim 7, characterized in that, The step of reverse-locating defects in the data production process based on the performance feedback data and dynamically adjusting the screening criteria of the data processing agent or the processing rules of the core processing pipeline includes: When the aforementioned factual errors are detected to occur frequently, it is determined to be a defect in the quality of the basic data, and a first optimization instruction is generated to correct the data cleaning rules or standardization transformation rules in the core processing pipeline. When frequent logical reasoning errors are detected or the target domain model is unable to handle subdivided scenarios, it is determined that the data logic chain or scenario coverage is insufficient, and a second optimization instruction is generated to adjust the annotation strategy of the high-quality question-answer pair or optimize the decision threshold of the data sorting agent.

9. The dataset construction method for model training according to claim 1, characterized in that, The method further includes: During the closed-loop iterative optimization process, the dataset version and corresponding model performance metrics for each round are recorded. Establish a mapping relationship between data versions and the capabilities of the target domain large model. By comparing and analyzing the impact of different dataset versions on the performance of the target domain large model, continuously update the preset knowledge base of the data processing agent.

10. A dataset construction device for model training, characterized in that, The device includes: The data sorting agent module is used to build a data sorting agent based on a large language model. The data sorting agent receives multi-source raw data and target domain business requirements, performs deep semantic analysis and vectorized matching on the multi-source raw data, and filters out highly relevant and effective data sets. The core pipeline processing module is used to input the highly correlated and effective data set into the automated core processing pipeline, and sequentially perform multimodal data standardization processing, high-quality question-answer pair construction and security quality enhancement processing to generate a standard target domain dataset for training large models in the target domain. The closed-loop iterative optimization module is used to train and evaluate the target domain large model based on the standard target domain dataset, collect the performance feedback data of the target domain large model on business tasks, and construct a closed-loop iterative optimization mechanism. The dynamic adjustment module is used to reverse locate the defects in the data production process based on the performance feedback data, and dynamically adjust the screening criteria of the data sorting agent or the processing rules of the core processing pipeline until the performance of the target domain large model meets the preset requirements.

Images