Domain vertical large language model fine-tuning dataset construction method based on multi-library linkage

Abstract

This invention discloses a method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, comprising the following steps: collecting structured business data, unstructured text data, and knowledge graph data to form a structured business database, an unstructured text database, and a knowledge graph database, respectively; establishing a multi-database linkage and association mechanism; establishing a dynamic priority mechanism to resolve conflicts according to priority when different data sources conflict; generating structured instruction question-and-answer samples using quadruples based on the multi-database linkage and association results; and using a large language model for diversified rewriting and quality filtering to construct a domain-specific large language model fine-tuning dataset. This invention effectively solves the technical problem of inconsistencies between historical data and current business rules by using data priority for conflict resolution; and significantly improves sample quality and generation efficiency by using quadruples to generate structured instruction question-and-answer samples.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage. Background Technology

[0002] With the rapid development of Large Language Model (LLM) technology, the demand for domain-specific large language models is growing across various industries. Domain-specific large language models can be applied to multiple scenarios such as intelligent customer service, professional consulting, content generation, and decision support, significantly improving service quality and work efficiency.

[0003] However, in the development of large language models for specific domains, instruction fine-tuning datasets often require high quality, comprehensive coverage, and a large sample size. Current methods of manually annotating datasets are costly and error-prone, making automated construction of instruction fine-tuning datasets a necessary technical approach. Existing automated technologies generally face the following pain points when constructing domain-specific datasets:

[0004] 1. Knowledge silos and single-dimensionality issues: Traditional methods often generate instructions based on single documents or isolated databases, making it difficult to simulate the complex cross-domain expert logic in reality. This results in the model's poor performance when handling comprehensive professional problems. Business data is scattered and stored in multiple independent data sources:

[0005] Structured business data types: stored in relational databases (such as MySQL, PostgreSQL), data warehouses, and other systems, mainly consisting of structured business records;

[0006] Unstructured text data type: Stored in document management systems, content management systems, etc., mainly consisting of unstructured text such as rules and specifications, operation guides, etc. In addition, semi-structured data such as business operation logs (usually in JSON, XML or plain text form) are also classified into this category and stored in document databases (such as MongoDB, ELK search engine).

[0007] Knowledge graph data type: Stored in graph databases (such as FalkorDB, Neo4j), organizing domain concepts and relationships in the form of a graph database;

[0008] Existing technical solutions typically use only a single data source for fine-tuning, resulting in one-sided model knowledge and an inability to form a complete domain knowledge system.

[0009] 2. Non-standardization issues in instruction generation: Existing instruction-guided methods rely heavily on fragmented and vague natural language prompts, lacking structured constraints on the generation process. This results in highly variable sample quality and insufficient professionalism. The main characteristics are strong subjectivity, poor repeatability, and unstable quality, making it difficult to meet the needs of large-scale domain-specific models for high-quality fine-tuning data.

[0010] 3. Potential for Multi-Source Data Conflicts: When multiple data sources are introduced, inconsistencies in information update frequency or professional terminology can easily lead to logically contradictory fine-tuning data. Without an automated resolution mechanism, this will severely compromise the model's knowledge consistency. Specific problems include: difficulties in semantic understanding and association between different data formats (e.g., associating business records in a database with rule specification texts); inconsistent information due to asynchronous data updates (e.g., mismatch between historical business records and new rule specifications after organizational restructuring); and a lack of a unified priority configuration mechanism, making it impossible to flexibly adjust the authority of data sources according to the characteristics of different domains.

[0011] 4. Lack of Domain-Specific Priority Configuration: Different domains have significantly different requirements for data source priority, and existing technologies lack configurable priority mechanisms.

[0012] In the field of social governance: dynamic information such as leadership appointments and organizational restructuring is mainly based on the rules and regulations published on the official website, which have a higher priority than historical appointment records in the structured database;

[0013] In the realm of enterprise operations: Business data such as customer orders and inventory quantities should be based on structured database records, which take precedence over rule-based specifications. For example, the status of a customer order (shipped / not shipped) should be based on the real-time records in the order database, rather than the planned shipping time in the contract text.

[0014] The aforementioned problems cause large models to learn outdated or incorrect knowledge associations, affecting the accuracy and practicality of the models.

[0015] 5. Low quality of fine-tuned samples: Existing fine-tuned datasets mainly rely on manual annotation, which suffers from high annotation costs, inconsistent annotation standards, and a lack of domain-specific knowledge support. More importantly, existing sample generation methods do not fully utilize the correlations between multi-source data, resulting in incomplete sample information, inaccurate knowledge citations, and incomplete sample coverage. Summary of the Invention

[0016] To address the aforementioned technical issues, this invention provides a method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, which features a simple algorithm and high-quality samples.

[0017] The technical solution of this invention to solve the above-mentioned technical problems is: a method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, comprising the following steps:

[0018] Step 1: Collect structured business data, unstructured text data, and knowledge graph data to form a structured business database, an unstructured text database, and a knowledge graph database, respectively;

[0019] Step 2: Establish a multi-database linkage mechanism;

[0020] Step 3: Establish a dynamic priority mechanism to resolve conflicts according to priority order when different data sources conflict.

[0021] Step 4: Based on the results of multi-database linkage and association, generate structured instruction question-and-answer samples using quadruple instructions;

[0022] Step 5: Use a large language model for diversified rewriting and quality filtering to build a domain-specific large language model fine-tuning dataset.

[0023] The above-described method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, in step 1, involves a structured business database responsible for recording business events. This structured business data describes business events, which refer to various business activities occurring within an organization, and are represented as follows:

[0024] M={id, type, content1, source_system, timestamp, quality_tag};

[0025] Where M represents the business event; id represents the unique identifier of the business event; type represents the type of the business event; content1 represents the content of the business event; source_system represents the data source system; timestamp represents the time when the business event occurred; and quality_tag represents the data quality tag.

[0026] The above-described method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, in step 1, involves an unstructured text database responsible for rule specifications and operational guidelines. The unstructured text data describes domain knowledge clauses, which refer to the specific knowledge content of the rule specifications in each domain, expressed as:

[0027] L={lid, name, identifier, content2, effective_date, source_system};

[0028] Where L represents the domain knowledge clause; lid is the unique identifier of the clause; name is the name of the knowledge base; identifier is the clause number or identifier; content2 is the clause content; effective_date is the effective date; and source_system is the data source system.

[0029] The above-described method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, in step 1, involves a knowledge relationship graph database responsible for modeling domain concepts and relationships. A knowledge relationship graph is a data structure that uses a graph database model to represent the relationships between domain knowledge, including organizational structure and standard operating procedures. The knowledge relationship graph is defined as follows:

[0030] G = (V, E, Attr);

[0031] Where G is the knowledge relationship graph; V is the set of organizational nodes, representing knowledge entities; E is the set of edges, representing knowledge relationships; Attr is the set of attributes of organizational nodes and edges; V and E constitute the organizational structure.

[0032] The above-mentioned method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage includes the following six specific linkage mechanisms in step 2:

[0033] Business event-rule specification association: Semantically match business events in the structured business database with relevant rule specifications in the unstructured text database;

[0034] Business Event-Organization Node Association: Mapping the executing entities of business events in the structured business database to organization nodes in the knowledge relationship graph database;

[0035] Rule Specifications - Organizational Structure Association: Align and validate the responsible parties of rule specifications in the unstructured text library with the organizational structure in the knowledge relationship graph database;

[0036] Business event-process step association: Establish associations between business events in the structured business database and the corresponding steps of standard operating procedures in the knowledge relationship graph database;

[0037] Rule Specification-Process Requirements Association: Associate the mandatory requirements of rule specifications in the unstructured text library with the checkpoints of standard operating procedures in the knowledge relationship graph database;

[0038] Organization Node-Organization Node Responsibility Association: This involves associating and mapping the responsibility descriptions of organization nodes in the knowledge graph database with those in the unstructured text database.

[0039] The specific process of step 3 in the above-mentioned method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage is as follows:

[0040] Step 31: Manually configure the priority order of different data sources, then proceed to Step 32;

[0041] Step 32: Perform conflict detection. If no conflict is detected, the configured priority order is used as the default data source, and then proceed to step 35. If different data sources conflict in their descriptions of the same item, proceed to step 33.

[0042] Step 33: Calculate the priority of different data sources and compare their priorities. If the priorities of different data sources are different, use the information from the higher priority data source in order of priority, and then proceed to step 35; if the priorities of different data sources are the same, proceed to step 34.

[0043] Step 34: Activate the logical consistency detection mechanism and perform logical consistency comparison based on the knowledge relationship graph. If there are relevant fact records in the knowledge relationship graph, select the data source that is consistent with the existing facts in the knowledge relationship graph. If there are no relevant fact records in the knowledge relationship graph, it is considered that the conflict is unresolved. Keep the original comparison result and process it manually. Then proceed to step 35.

[0044] Step 35: Log the conflict detection and handling results.

[0045] In the above method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, step 4 involves using four-tuple instructions as structured data extracted from the multi-database linkage results to generate instruction question-and-answer samples.

[0046] TupleQA=(Qelem,Lbasis,Oinfo,Pinfo);

[0047] Among them, TupleQA is a four-tuple instruction; Qelem is the question element, including question type, content, query subject, and application scenario; Lbasis is the knowledge clause, including knowledge name, clause identifier, and clause content; Oinfo is the organizational information, including organizational structure name, organizational node name, organizational node responsibility description, contact information, and work location; Pinfo is the anonymized information of the associated responsible person, including the responsible person's name, position, contact information, and scope of responsibilities.

[0048] The specific process of step 4 in the above-mentioned method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage is as follows:

[0049] First, knowledge is deconstructed based on the results of multi-database linkage and correlation. Four-tuple instructions with business logic support are extracted from the results of multi-database linkage and correlation. The four-tuple instructions include problem elements, knowledge clauses, organizational information, and information of the related responsible persons after anonymization.

[0050] Then, based on the task type, the quadruple instruction is injected into the preset instruction template library to select an instruction template. The instruction template library includes consultation templates, guidance templates, query templates, and process templates.

[0051] Next, instruction question-and-answer samples are generated by automatically matching instruction templates;

[0052] Finally, after labeling the task type and data source, output the instruction question and answer sample.

[0053] The specific process of step 5 in the above-mentioned method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage is as follows:

[0054] Step 51: For the instruction question-and-answer samples, design prompt words for the large language model to guide the large language model to rewrite the language style in multiple dimensions while keeping the core semantic logic unchanged;

[0055] Step 52: The Large Language Model (LLM) generates five diverse variants through an automated generation mechanism, including: colloquial variants that simulate the language of non-professional users, formal variants that conform to technical specifications, concise variants that focus on the core conclusions, detailed variants that contain a complete chain of arguments, and contextual variants that introduce preconditions.

[0056] Step 53: Perform quality assessment on all generated variants by decomposing the variants into multi-dimensional evaluation metrics, including information completeness, rule accuracy, consistency of rights and responsibilities, language fluency, and task matching. Calculate a comprehensive score for each variant. Only variants with a comprehensive score greater than or equal to 0.8 are considered high-quality samples and retained. For variants with a comprehensive score below 0.8, automatic elimination is performed, ultimately obtaining the domain-specific large language model fine-tuning dataset.

[0057] The beneficial effects of this invention are as follows: This invention breaks down data silos and achieves the organic integration of structured business data, unstructured text data, and knowledge relationship graph data; it effectively solves the technical problem of inconsistency between historical data and current business rules by using data priority for conflict resolution; and it significantly improves sample quality and generation efficiency by using quadruple instructions to generate structured instruction question-and-answer samples. Attached Figure Description

[0058] Figure 1 This is the overall flowchart of the present invention.

[0059] Figure 2 This is a flowchart of the data priority conflict resolution process of the present invention.

[0060] Figure 3 This is a flowchart for generating instruction question-and-answer samples for the present invention.

[0061] Figure 4 This is a flowchart illustrating the sample diversification generation and quality filtering process of the present invention. Detailed Implementation

[0062] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0063] like Figure 1 As shown, the method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage includes the following steps:

[0064] Step 1: Collect structured business data, unstructured text data, and knowledge graph data to form a structured business database, an unstructured text database, and a knowledge graph database, respectively.

[0065] The structured business database is responsible for recording business events. Structured business data describes business events, which refer to various business activities that occur within the organization, and are represented as follows:

[0066] M={id, type, content1, source_system, timestamp, quality_tag};

[0067] Where M represents the business event; id represents the unique identifier of the business event; type represents the type of the business event; content1 represents the content of the business event; source_system represents the data source system; timestamp represents the time when the business event occurred; and quality_tag represents the data quality tag.

[0068] The unstructured text library is responsible for rules, specifications, and operational guidelines. Unstructured text data describes domain knowledge clauses, which refer to the specific knowledge content of rules and specifications within each domain, represented as:

[0069] L={lid, name, identifier, content2, effective_date, source_system};

[0070] Where L represents the domain knowledge clause; lid is the unique identifier of the clause; name is the name of the knowledge base; identifier is the clause number or identifier; content2 is the clause content; effective_date is the effective date; and source_system is the data source system.

[0071] Knowledge graph databases are responsible for modeling domain concepts and relationships. A knowledge graph is a data structure that uses a graph database model to represent the relationships between domain knowledge, including organizational structure and standard operating procedures. A knowledge graph is defined as follows:

[0072] G = (V, E, Attr);

[0073] Wherein, G is the knowledge relationship graph; V is the set of organizational nodes, representing knowledge entities; E is the set of edges, representing knowledge relationships; Attr is the set of attributes of organizational nodes and edges; V and E constitute the organizational structure; the knowledge relationship graph is stored in a knowledge relationship graph database, supporting complex relationship queries and reasoning.

[0074] Step 2: Establish a multi-database linkage mechanism. The multi-database linkage mechanism includes the following six specific linkage mechanisms:

[0075] Business event-rule specification association: Semantically match business events in the structured business database with relevant rule specifications in the unstructured text database;

[0076] Business Event-Organization Node Association: Mapping the executing entities of business events in the structured business database to organization nodes in the knowledge relationship graph database;

[0077] Rule Specifications - Organizational Structure Association: Align and validate the responsible parties of rule specifications in the unstructured text library with the organizational structure in the knowledge relationship graph database;

[0078] Business event-process step association: Establish associations between business events in the structured business database and the corresponding steps of standard operating procedures in the knowledge relationship graph database;

[0079] Rule Specification-Process Requirements Association: Associate the mandatory requirements of rule specifications in the unstructured text library with the checkpoints of standard operating procedures in the knowledge relationship graph database;

[0080] Organization Node-Organization Node Responsibility Association: This involves associating and mapping the responsibility descriptions of organization nodes in the knowledge graph database with those in the unstructured text database.

[0081] Step 3: Establish a dynamic priority mechanism to resolve conflicts according to priority order when different data sources conflict.

[0082] like Figure 2 As shown, the specific process of step 3 is as follows:

[0083] Step 31: Manually configure the priority order of different data sources. The priority is sorted according to the priority parameters configured by the user. Different priority parameters can be configured for different business scenarios, thereby improving the quality of each batch of fine-tuned data samples through simple dynamic configuration parameters.

[0084] Step 32: Perform conflict detection. If no conflict is detected, the configured priority order is used as the default data source, and then proceed to step 35. If different data sources conflict in their descriptions of the same item, proceed to step 33.

[0085] Step 33: Calculate the priority of different data sources and compare their priorities. If the priorities of different data sources are different, use the information from the higher priority data source in order of priority, and then proceed to step 35; if the priorities of different data sources are the same, proceed to step 34.

[0086] Step 34: Activate the logical consistency detection mechanism and perform logical consistency comparison based on the knowledge relationship graph. If there are relevant fact records in the knowledge relationship graph, select the data source that is consistent with the existing facts in the knowledge relationship graph. If there are no relevant fact records in the knowledge relationship graph, it is considered that the conflict is unresolved. Keep the original comparison result and process it manually. Then proceed to step 35.

[0087] Step 35: Log the conflict detection and handling results.

[0088] Step 4: Based on the results of multi-database linkage, generate structured instruction question-and-answer samples using quadruple instructions.

[0089] Quadruple instructions are structured data extracted from the results of multi-database linkage and association, used to generate instruction question-and-answer samples:

[0090] TupleQA=(Qelem,Lbasis,Oinfo,Pinfo);

[0091] Among them, TupleQA is a four-tuple instruction; Qelem is the question element, including question type, content, query subject, and application scenario; Lbasis is the knowledge clause, including knowledge name, clause identifier, and clause content; Oinfo is the organizational information, including organizational structure name, organizational node name, organizational node responsibility description, contact information, and work location; Pinfo is the anonymized information of the associated responsible person, including the responsible person's name, position, contact information, and scope of responsibilities.

[0092] like Figure 3 As shown, the specific process of step 4 is as follows:

[0093] First, knowledge is deconstructed based on the results of multi-database linkage and correlation. Four-tuple instructions with business logic support are extracted from the results of multi-database linkage and correlation. The four-tuple instructions include problem elements, knowledge clauses, organizational information, and information of the related responsible persons after anonymization. They cover the facts of task triggering, the compliance basis of processing, the entity dimensions involved, and the boundaries of execution authority.

[0094] Then, based on the task type, the quadruple instruction is injected into the preset instruction template library to select an instruction template. The instruction template library includes consultation templates, guidance templates, query templates and process templates to promote the rigor of the sample in logical structure and the professionalism of the vertical field.

[0095] Next, instruction question-and-answer samples are generated by automatically matching instruction templates;

[0096] Finally, after labeling the task type and data source, output the instruction question and answer sample.

[0097] Step 5: Use a large language model for diversified rewriting and quality filtering to build a domain-specific large language model fine-tuning dataset.

[0098] like Figure 4 As shown, the specific process of step 5 is as follows:

[0099] Step 51: For the instruction question-and-answer samples, design prompt words for the large language model to guide the large language model to rewrite the language style in multiple dimensions while keeping the core semantic logic unchanged;

[0100] Step 52: The Large Language Model (LLM) generates five diverse variants through an automated generation mechanism, including: colloquial variants that simulate the language of non-professional users, formal variants that conform to technical specifications, concise variants that focus on core conclusions, detailed variants that contain complete argument chains, and contextual variants that introduce preconditions. This simulation of multimodal contexts aims to significantly enhance the generalization ability of the fine-tuned Large Language Model in different interactive scenarios.

[0101] Step 53: Perform quality assessment on all generated variants by decomposing the variants into multi-dimensional evaluation metrics, including information completeness, rule accuracy, consistency of rights and responsibilities, language fluency, and task matching. Calculate a comprehensive score for each variant. Only variants with a comprehensive score greater than or equal to 0.8 are considered high-quality samples and retained. For variants with a comprehensive score below 0.8, automatic elimination is performed, ultimately obtaining the domain-specific large language model fine-tuning dataset.

[0102] Example 1

[0103] Application scenarios: Construct a large language model fine-tuning dataset in the field of social governance for use in scenarios such as intelligent customer service for business service hotlines, community service consultation, and policy interpretation.

[0104] Step 1: Collect structured business data (service request records, stored in a MySQL database), unstructured text data (community rules and specifications documents, stored in a file system), and knowledge graph data (organizational structure of community services, stored in a FalkorDB database), which respectively constitute the structured business database, the unstructured text database, and the knowledge graph database.

[0105] Step 2: Establish a multi-database linkage mechanism.

[0106] Business Event - Rule and Specification Association: Extract relevant rules and specifications (such as Article 35 of the "Property Management Regulations") from service request records, perform semantic matching with the clauses in the community rules and specifications document, and establish a mapping relationship between service request records and rules and specifications;

[0107] Business event-organization node association: Extract relevant department information (such as community service center) from service request records, query the responsibilities and permissions of the department in the organizational structure, and establish the correspondence between service request records and departments;

[0108] Rule and specification - organizational structure association: Extract responsible department information (such as property service department) from the rule and specification, align and verify it with the departments in the organizational structure, and establish a consistent association between the rule and specification and the organizational structure;

[0109] Business event-process step association: Extract the triggered business process type (such as maintenance process) from the service request record, match it with the corresponding step in the standard operating procedure in the organizational structure, and establish an execution association between the service request record and the process step;

[0110] Association of Rules and Specifications with Process Requirements: Extract mandatory requirements and time limits (such as completion within 5 working days) from the rules and specifications, and associate them with the checkpoints and time limits of the standard operating procedures in the organizational structure to establish a consistent mapping between the rules and specifications and the process requirements;

[0111] Organization Node-Organization Node Responsibility Association: A relationship mapping is established between organization nodes in the knowledge relationship graph database and the responsibility descriptions of organization nodes in the unstructured text library.

[0112] Step 3: Establish a dynamic priority mechanism to resolve conflicts according to priority order when different data sources conflict.

[0113] Scenario: A historical service request record (structured business database) shows that community maintenance is the responsibility of the facilities management office, but the job description of the property service office in the unstructured text database includes community maintenance service items, and the standard operating procedures of the property service office in the knowledge relationship graph database include the community maintenance process.

[0114] Processing flow:

[0115] First, manually configure the priority order parameters for different data sources:

[0116] Structured business data (service request records): High update frequency (daily increase), basic priority = 2.5;

[0117] Knowledge graph data (organizational structure): updates are relatively slow, but it has strong logical importance, with a priority of 2.5;

[0118] Unstructured text data (community rules and specifications documents): Priority is 2 if it is the latest version, based on the date of its creation.

[0119] Second, conflict detection was conducted: it was found that the department information (facilities management office) in the service request record was inconsistent with the job description (community maintenance service items) of the property service office in the unstructured text database. The standard operating procedures of the property service office in the knowledge relationship graph database include the community maintenance process.

[0120] Third, perform a priority comparison: Service request record priority = 2.5, knowledge relationship graph priority = 2.5, the priorities are the same, proceed to step 34;

[0121] Fourth, initiate a logical consistency detection mechanism, using the knowledge relationship graph as a benchmark to compare logical consistency and check whether the community maintenance handled by the facility management office is consistent with the community maintenance process of the property service office in the knowledge relationship graph; if there are relevant fact records in the knowledge relationship graph, select the data source consistent with the existing facts in the knowledge relationship graph; if there are no relevant fact records in the knowledge relationship graph, it is considered that the conflict is unresolved, and the original comparison result is maintained and manually processed.

[0122] Fifth, logging: Logs recording the results of conflict detection and handling, including conflict fields, priority of each data source, logical consistency detection results, data source used, etc., for subsequent auditing and traceability.

[0123] Step 4: Based on the results of multi-database linkage and association, generate structured instruction question-and-answer samples using quadruple instructions;

[0124] Step 5: Use a large language model for diversified rewriting and quality filtering to build a domain-specific large language model fine-tuning dataset.

[0125] Example 2

[0126] Application scenarios: Construct large language model fine-tuning datasets for large enterprise operations, used in scenarios such as intelligent operations assistants, decision support, and process automation.

[0127] Step 1: Collect structured business data (business transaction events, stored in a MySQL database), unstructured text data (operation manuals and regulations, stored in a file system), and knowledge graph data (organizational structure and approval processes, stored in a FalkorDB database), which respectively constitute the structured business database, the unstructured text database, and the knowledge graph database.

[0128] For example, the business transaction event is: Order number PO20230815, order amount 800,000 yuan, supplier A company, current status: pending payment;

[0129] The operation manual and regulations stipulate that: Article 8 of the "Enterprise Procurement Management System" states that "a single purchase of more than 500,000 yuan must be approved by the general manager," and Article 12 states that "warehousing acceptance must be completed before payment."

[0130] The organizational structure and approval process are as follows: Zhang is in charge of procurement, Wang is in charge of finance, and the approval process is: procurement application → general manager approval → finance payment.

[0131] Step 2: Establish a multi-database linkage mechanism.

[0132] Business Event-Rules and Regulations Association: Extract relevant rules and regulations identifiers (such as Article 15 of the Company Law) from business transaction event records, perform semantic matching with the clauses in the operation manual and system regulations, and establish a mapping relationship between business transaction events and operation manual and system regulations;

[0133] Business Event-Organization Node Association: Extract the organizational nodes (such as purchasing or sales departments) involved in business transaction events, query the responsibilities and permissions of the organizational nodes in the knowledge relationship graph, and establish the correspondence between business transaction events and organizational nodes;

[0134] Rule and Regulation - Organizational Structure Association: Extract responsible department information (such as the finance department) from operation manuals and regulations, align and verify it with the departments in the knowledge relationship diagram, and establish a consistent association between rule and regulation and organizational structure;

[0135] Business event-process step association: Extract the triggered business process type (such as procurement process or sales process) from business transaction events, match it with the approval process operation steps in the knowledge relationship diagram, and establish the execution association between business events and process steps;

[0136] Relationship between rules and regulations and process requirements: Extract mandatory requirements and time limits (such as completion within 3 working days) from operation manuals and regulations, and associate them with checkpoints and time limits of the approval process in the knowledge relationship diagram to establish a consistent mapping between rules and regulations and process requirements;

[0137] Organization Node-Organization Node Responsibility Association: This involves mapping the responsibility descriptions of organization nodes in the knowledge graph database to those in the unstructured text database, thus establishing the association between organization nodes and their responsibilities.

[0138] Step 3: Establish a dynamic priority mechanism to resolve conflicts according to priority order when different data sources conflict.

[0139] Scenario: A historical business transaction record shows that "purchase orders are approved by the purchasing department", but the knowledge relationship diagram clearly states that this matter is the responsibility of the "supply chain management department" (renamed after organizational restructuring in 2024).

[0140] Processing flow:

[0141] First, manually configure the priority order of different data sources:

[0142] Structured business data (business transaction events): High update frequency (daily increase), basic priority = 3;

[0143] Knowledge graph data (organizational structure and approval process): updates are relatively slow, but it has strong logical importance, with a basic priority of 2.5;

[0144] Unstructured text data (operation manuals and regulations, including descriptions of responsibilities of organizational nodes): Priority is 2 based on the date of formulation, if it is the latest version;

[0145] Second, conduct conflict detection by comparing the department information (procurement department) in the business transaction records with the department responsibilities (supply chain management department) in the knowledge relationship diagram to identify inconsistencies;

[0146] Third, prioritize the information from the structured business data to determine which information to use (because the priority of structured business data is greater than that of knowledge graph data).

[0147] Fourth, logging: Logs recording the results of conflict detection and handling, including conflict fields, data sources used, version numbers, and other information, for subsequent auditing and traceability.

[0148] Step 4: Based on the results of multi-database linkage, generate structured instruction question-and-answer samples using four-tuple instructions. The four-tuple instructions are: Qelem for querying the payment status and approval requirements of a purchase order; Lbasis for Articles 8 and 12 of the procurement management system; Oinfo for the procurement department and finance department; and Pinfo for Zhang (the person in charge of procurement) and Wang (the person in charge of finance).

[0149] The sample Q&A is as follows: Standard Question: Inquire about the payment status and approval process requirements for purchase order PO20230815; Standard Answer: [This order is currently in a pending payment status. According to Article 8 of the "Enterprise Procurement Management System," purchases exceeding 500,000 yuan require approval from the General Manager; Article 12 stipulates that warehousing acceptance must be completed before payment. Zhang, the person in charge of procurement, is responsible for order follow-up, and Wang, the person in charge of finance, is responsible for payment processing.]

[0150] Step 5: Use a large language model for diversified rewriting and quality filtering to build a domain-specific large language model fine-tuning dataset.

[0151] Step 51: Input the standard sample generated based on the template into the Large Language Model (LLM). Based on the instruction question-and-answer sample obtained in Step 4, design prompt words for the Large Language Model. The prompt words should meet the following requirements: 1. Keep the core information unchanged (department, office, rules, responsible person); 2. Change the questioning angle (conversational / formal / concise / detailed); 3. Change the order of information; 4. Add reasonable background information; 5. Simulate different user groups (financial responsible person / purchasing responsible person / auditing responsible person).

[0152] Step 52: Generate 5 different question variations as follows:

[0153] A colloquial variation: "Could you check if purchase order PO20230815 has been paid? What's the current status of the process, and who should I contact?"

[0154] Formal variant: "Please check the current payment status of purchase order PO20230815 and explain the approval process requirements and contact information of the responsible person for this order in accordance with the 'Enterprise Procurement Management System'."

[0155] A concise variation: "Who needs to approve PO20230815? Who is responsible?"

[0156] Detailed variant: "As a financial auditor, you need to verify whether the payment process for purchase order PO20230815 (amount 800,000 yuan) is compliant. Please provide: 1) the current payment status; 2) the specific requirements of Articles 8 and 12 of the 'Enterprise Procurement Management System'; 3) the names and contact information of the persons responsible for each step of the approval process."

[0157] Context variation: "I am a new purchasing agent and would like to understand the approval process and responsible person for purchase orders over 500,000 in our company. Please explain using purchase order PO20230815 as an example."

[0158] Step 53: Design multi-dimensional evaluation indicators:

[0159] 1. Information Completeness (Weight 0.20): Check if all required fields are complete (responsible department, handling office, responsible person, contact number, office address, processing time limit, list of required materials);

[0160] 2. Rule accuracy (weight 0.25): Verify whether the description of organizational node responsibilities and the reference to approval processes are correct (rule name, clause number, clause content);

[0161] 3. Consistency of authority and responsibility (weight 0.25): Check whether the description of the responsibilities of the organizational nodes is consistent with the knowledge relationship diagram;

[0162] 4. Language fluency (weight 0.15): The naturalness of language is assessed using LLM;

[0163] 5. Task matching degree (weight 0.15): Check whether the output meets the requirements of the task type (e.g., consultation type should include the responsible person information, guidance type should include process steps, etc.).

[0164] Overall score calculation formula:

[0165] Score = 0.20 × Completeness + 0.25 × LawAccuracy + 0.25 × OfficeConsistency + 0.15 × Fluency + 0.15 × TaskAlignment; Score represents the overall score, Completeness represents the completeness of information, LawAccuracy represents the accuracy of rules, OfficeConsistency represents the consistency of authority and responsibility, Fluency represents the fluency of language, and TaskAlignment represents the task matching degree.

[0166] Filtering rule: Retain high-quality samples with a score ≥ 0.8.

[0167] The technical effects are compared in Table 1:

[0168] Table 1

[0169]

[0170] Note: Surface accuracy refers to the accuracy of samples in terms of surface features such as syntax and format; Comprehensive quality index = surface accuracy × complexity weight, where complexity weight = cross-database linkage capability coefficient × logical complexity coefficient, reflecting the actual contribution of samples to the improvement of model capabilities.

[0171] This invention significantly improves sample generation efficiency, reducing manual editing time by 95%. Although manual annotation has slightly higher surface accuracy, its overall quality index is far lower than that of this invention. The high surface accuracy (97.3%) of manual annotation only reflects the correctness of samples in terms of grammar, format, and other superficial features, but lacks cross-database linkage capabilities and complex logical structures, thus contributing little to improving the model's specialization capabilities. Although this invention has slightly lower surface accuracy (95.7%), it features three-database linkage and high logical complexity, with an overall quality index 1.825 times that of manual annotation, effectively training the complex reasoning and conflict resolution capabilities of large-scale models in specific domains.

Claims

1. A method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage, characterized in that: Includes the following steps: Step 1: Collect structured business data, unstructured text data, and knowledge graph data to form a structured business database, an unstructured text database, and a knowledge graph database, respectively; Step 2: Establish a multi-database linkage mechanism; Step 3: Establish a dynamic priority mechanism to resolve conflicts according to priority order when different data sources conflict. Step 4: Based on the results of multi-database linkage and association, generate structured instruction question-and-answer samples using quadruple instructions; Step 5: Use a large language model for diversified rewriting and quality filtering to build a domain-specific large language model fine-tuning dataset.

2. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage according to claim 1, characterized in that, In step 1, the structured business database is responsible for recording business events. The structured business data describes business events, which refer to various business activities that occur within the organization, and are represented as follows: M={id, type, content1, source_system, timestamp, quality_tag}; Where M represents the business event; id represents the unique identifier of the business event; type represents the type of the business event; content1 represents the content of the business event; source_system represents the data source system; timestamp represents the time when the business event occurred; and quality_tag represents the data quality tag.

3. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage as described in claim 2, characterized in that, In step 1, the unstructured text library is responsible for the rule specifications and operational guidelines. The unstructured text data describes domain knowledge clauses, which refer to the specific knowledge content of the rule specifications in each domain, expressed as: L={lid, name, identifier, content2, effective_date, source_system}; Where L represents the domain knowledge clause; lid is the unique identifier of the clause; name is the name of the knowledge base; identifier is the clause number or identifier; content2 is the clause content; effective_date is the effective date; and source_system is the data source system.

4. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage according to claim 3, characterized in that, In step 1, the knowledge relationship graph database is responsible for domain concept and relationship modeling. A knowledge relationship graph is a data structure that uses a graph database model to represent the relationships between domain knowledge, including organizational structure and standard operating procedures. The knowledge relationship graph is defined as follows: G = (V, E, Attr); Where G is the knowledge relationship graph; V is the set of organizational nodes, representing knowledge entities; E is the set of edges, representing knowledge relationships; Attr is the set of attributes of organizational nodes and edges; V and E constitute the organizational structure.

5. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage according to claim 4, characterized in that, In step 2, the multi-database linkage and association mechanism includes the following six specific association mechanisms: Business event-rule specification association: Semantically match business events in the structured business database with relevant rule specifications in the unstructured text database; Business Event-Organization Node Association: Mapping the executing entities of business events in the structured business database to organization nodes in the knowledge relationship graph database; Rule Specifications - Organizational Structure Association: Align and validate the responsible parties of rule specifications in the unstructured text library with the organizational structure in the knowledge relationship graph database; Business event-process step association: Establish associations between business events in the structured business database and the corresponding steps of standard operating procedures in the knowledge relationship graph database; Rule Specification-Process Requirements Association: Associate the mandatory requirements of rule specifications in the unstructured text library with the checkpoints of standard operating procedures in the knowledge relationship graph database; Organization Node-Organization Node Responsibility Association: This involves associating and mapping the responsibility descriptions of organization nodes in the knowledge graph database with those in the unstructured text database.

6. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage according to claim 5, characterized in that, The specific process of step 3 is as follows: Step 31: Manually configure the priority order of different data sources, then proceed to Step 32; Step 32: Perform conflict detection. If no conflict is detected, use the configured priority order as the default data source, and then proceed to step 35. If different data sources provide conflicting descriptions of the same item, proceed to step 33; Step 33: Calculate the priority of different data sources and compare their priorities. If the priorities of different data sources are different, use the information from the higher priority data source in order of priority, and then proceed to step 35; if the priorities of different data sources are the same, proceed to step 34. Step 34: Activate the logical consistency detection mechanism and perform logical consistency comparison based on the knowledge relationship graph. If there are relevant fact records in the knowledge relationship graph, select the data source that is consistent with the existing facts in the knowledge relationship graph. If there are no relevant fact records in the knowledge relationship graph, it is considered that the conflict is unresolved. Keep the original comparison result and process it manually. Then proceed to step 35. Step 35: Log the conflict detection and handling results.

7. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage as described in claim 6, characterized in that, In step 4, the four-tuple instruction is structured data extracted from the multi-database linkage results and is used to generate instruction question-and-answer samples: TupleQA=(Qelem,Lbasis,Oinfo,Pinfo); Among them, TupleQA is a four-tuple instruction; Qelem is the question element, including question type, content, query subject, and application scenario; Lbasis is the knowledge clause, including knowledge name, clause identifier, and clause content; Oinfo is the organizational information, including organizational structure name, organizational node name, organizational node responsibility description, contact information, and work location; Pinfo is the anonymized information of the associated responsible person, including the responsible person's name, position, contact information, and scope of responsibilities.

8. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage according to claim 7, characterized in that, The specific process of step 4 is as follows: First, knowledge is deconstructed based on the results of multi-database linkage and correlation. Four-tuple instructions with business logic support are extracted from the results of multi-database linkage and correlation. The four-tuple instructions include problem elements, knowledge clauses, organizational information, and information of the related responsible persons after anonymization. Then, based on the task type, the quadruple instruction is injected into the preset instruction template library to select an instruction template. The instruction template library includes consultation templates, guidance templates, query templates, and process templates. Next, instruction question-and-answer samples are generated by automatically matching instruction templates; Finally, after labeling the task type and data source, output the instruction question and answer sample.

9. The method for constructing a domain-specific large language model fine-tuning dataset based on multi-database linkage as described in claim 8, characterized in that, The specific process of step 5 is as follows: Step 51: For the instruction question-and-answer samples, design prompt words for the large language model to guide the large language model to rewrite the language style in multiple dimensions while keeping the core semantic logic unchanged; Step 52: The Large Language Model (LLM) generates five diverse variants through an automated generation mechanism, including: colloquial variants that simulate the language of non-professional users, formal variants that conform to technical specifications, concise variants that focus on the core conclusions, detailed variants that contain a complete chain of arguments, and contextual variants that introduce preconditions. Step 53: Perform quality assessment on all generated variants by decomposing the variants into multi-dimensional evaluation metrics, including information completeness, rule accuracy, consistency of rights and responsibilities, language fluency, and task matching. Calculate a comprehensive score for each variant. Only variants with a comprehensive score greater than or equal to 0.8 are considered high-quality samples and retained. For variants with a comprehensive score below 0.8, automatic elimination is performed, ultimately obtaining the domain-specific large language model fine-tuning dataset.

Images