A Knowledge Graph-Based Method for Generating High-Quality Command Data in Shipbuilding

Abstract

This invention provides a method for generating high-quality instruction data in the shipbuilding industry based on knowledge graphs, including: (1) multi-source text acquisition and preprocessing; (2) knowledge extraction and graph construction; (3) knowledge point understanding, evaluation, and labeling; (4) constrained subgraph extraction; (5) generation of multi-type instruction-response pairs; (6) data source monitoring and incremental graph update; (7) incremental update and review of instruction-response pairs; and (8) multi-dimensional quality assessment and output of instruction data. The purpose of this invention is to solve the problems of high training data construction cost, high illusion rate, low coverage, and poor timeliness during the fine-tuning of large language models in the shipbuilding industry.

Description

Technical Field

[0001] This invention relates to the interdisciplinary fields of artificial intelligence and shipbuilding industry, and in particular to a method for generating high-quality instruction data in the field of shipbuilding based on knowledge graphs. Background Technology

[0002] Supervised fine-tuning (SFT) of large language models (LLM) is the core approach for them to acquire domain expertise. However, this process relies heavily on large-scale, high-quality supervised training data. Such data requires manual annotation by domain experts, which results in high construction costs and long cycles, making it difficult to adapt to the needs of rapid updates in professional knowledge.

[0003] Existing methods for automatically synthesizing training data using LLM have significant drawbacks in knowledge-intensive fields such as shipbuilding: they are prone to factual errors (caused by model illusions), have insufficient coverage of long-tail knowledge, cannot accurately express the multi-step relationships in complex processes, and have a templated question-and-answer structure with a lack of diversity, which may limit the performance of the model after fine-tuning and make it difficult to meet the requirements of high-reliability scenarios.

[0004] The shipbuilding industry possesses a vast amount of specialized knowledge, which is scattered across various sources (including unstructured documents such as design documents, process specifications, industry standards, and research papers). This fragmented knowledge system makes it difficult to utilize systematically. While existing technologies can assist in constructing knowledge graphs for the shipbuilding industry, they lack a mechanism for automatically generating instruction-based question-and-answer training data based on these graphs. Furthermore, the knowledge graphs lack dynamic expansion capabilities and cannot incorporate the latest industry knowledge in a timely manner. Summary of the Invention

[0005] This invention provides a method for generating high-quality instruction data in the shipbuilding industry based on knowledge graphs. The aim is to solve the problems of high cost of training data construction, high illusion rate, low coverage, and poor timeliness in the process of fine-tuning large language models in the shipbuilding industry.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A knowledge graph-based method for generating high-quality instruction data in the shipbuilding industry is proposed to provide high-quality instruction data for supervised fine-tuning training of large-scale language models in a specific domain. The method includes:

[0008] (1) Collect multi-source heterogeneous text data in a specific field, preprocess the multi-source heterogeneous text data, and obtain standardized text corpus;

[0009] (2) Automatically extract key domain entities and their semantic relationships from the standardized text corpus, merge duplicate entities and relationships, resolve conflicting information using confidence level or manual verification, and construct an extensible domain knowledge graph; the domain knowledge graph consists of a set of key entities and a set of relationships;

[0010] (3) Use the training model to evaluate the understanding of each knowledge point in the domain knowledge graph and calculate the understanding loss of each knowledge point; mark the weak knowledge points and the mastered knowledge points according to the understanding loss threshold;

[0011] (4) Using the relationships in the domain knowledge graph as starting edges, and based on the understanding loss, depth constraint and length constraint, a specified traversal strategy is used to extract the subgraph to obtain the target subgraph;

[0012] (5) Based on the structural features of the target subgraph, automatically generate three types of instruction-response pairs: atomic, aggregate, and multi-hop.

[0013] (6) Monitor the updates and changes of the data source, repeat the preprocessing and knowledge extraction process of steps (1)-(2) for the newly added or modified text data, and incorporate the incremental knowledge into the domain knowledge graph to form a differential list containing new relationships, modified relationships and invalid relationships.

[0014] (7) For the new and modified relationships in the difference list, repeat steps (3)-(5) to generate new instruction-response pairs, mark or delete the original instruction-response pairs corresponding to the failed relationships, and conduct quality audits on the new instruction-response pairs;

[0015] (8) Perform multi-dimensional quality assessment on all instruction-response pairs, filter or regenerate substandard samples, and finally output a qualified high-quality supervised fine-tuning training dataset.

[0016] In this specification, the multi-source heterogeneous text data specifically includes internal enterprise documents, industry standards and specifications, published papers, patent documents, professional evaluation standards, design specifications, and process procedures in specific fields; the preprocessing specifically includes: format parsing and plain text extraction of the multi-source heterogeneous text data, removing noise such as headers and footers, duplicate page numbers, and invalid symbols, segmenting sentences or paragraphs, and standardizing synonyms and abbreviations.

[0017] In this specification, the key entities in the domain mentioned in step (2) include component class, process flow class, material class and standard specification class; the semantic relationships include component-function, component-composition, process-step, process-materials used, process-standards followed, component-applicable specifications; the construction of the domain knowledge graph also includes introducing a domain ontology model, mapping the extracted entities to predefined categories, limiting the available relational patterns, and then writing the entities and relations into a graph database to achieve unified storage and query.

[0018] In this specification, the specific process of understanding assessment in step (3) is as follows: each relation in the domain knowledge graph is converted into a natural language statement, multiple restatements and negations of the statement are generated using a synthesizer model, yes / no judgment prompts are constructed and input into the training model, the judgment confidence of the training model is obtained, the relation confidence and understanding loss of the relation are calculated based on the confidence, and when the understanding loss is higher than a preset threshold, it is marked as a weak knowledge point; otherwise, it is marked as a mastered knowledge point.

[0019] In this specification, the depth constraint in step (4) is to limit the maximum number of hops in the subgraph extraction, and the length constraint is to limit the upper limit of the premise text length corresponding to the subgraph; the specified traversal strategy includes three types: the strategy of prioritizing the selection of edges with high understanding loss, the strategy of prioritizing the selection of edges with low understanding loss, and the strategy of uniformly and randomly selecting edges, and a single strategy or a combination strategy is selected according to the training requirements.

[0020] In this specification, the atomic instruction-response pair described in step (5) is generated based on a single-sided or single-node subgraph to construct a question and answer for a single knowledge point; the aggregated instruction-response pair is generated based on a subgraph containing multiple relationships within the same topic domain, and after restating, compressing and summarizing the relationships through a synthesizer model to form a natural language answer, the corresponding question is generated in reverse; the multi-hop instruction-response pair is generated based on a subgraph spanning multiple relationships to construct a question requiring multi-step reasoning, and the answer depends on the combination of multiple knowledge chains.

[0021] In this specification, the updates and changes to the data source in step (6) specifically include text updates corresponding to the addition of new specifications, modification of processes, and introduction of new materials in a specific field; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed parts are supplemented or corrected; the quality audit and the multi-dimensional quality assessment in step (8) adopt the same standard.

[0022] In this specification, the multi-dimensional quality assessment in step (8) includes language dimension, knowledge dimension and structural dimension. The language dimension assesses grammatical correctness, terminology standardization and clarity of expression. The knowledge dimension assesses the consistency between the answer and the facts in the knowledge graph and the absence of illusions. The structural dimension assesses the distribution of question types, the distribution of difficulty levels and the repetition rate. The quality assessment also includes a comprehensive evaluation, which combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on closed-book knowledge-intensive tasks.

[0023] In this specification, when generating the instruction-response pairs, language diversity is enhanced in the following ways: multiple template questions are designed for each type of semantic relationship, and the template questions are rewritten in multiple rounds using a large language model to form multiple ways of asking questions that are semantically equivalent but express different meanings. In aggregated and multi-hop instruction-response pairs, multiple knowledge points are merged into the same question to increase the difficulty of comprehensive understanding.

[0024] In this specification, the specific field is the shipbuilding field. The domain knowledge graph constructed in step (2) includes the welding process-step-materials-inspection standard subdomain. The instruction-response pairs generated in step (5) involve atomic, aggregate and multi-hop instruction-response pairs related to welding method operation steps, weld quality judgment standards and defect handling measures.

[0025] In summary, the present invention has at least the following beneficial effects:

[0026] It significantly reduces the cost of manual annotation, eliminates the need to rely on a large number of experts to manually construct training data, and enables the automated synthesis of high-quality SFT data.

[0027] Improve the core quality of training data: By using knowledge graph structured modeling, significantly reduce the data illusion rate, improve the systematicness and completeness of knowledge coverage, and cover the correlation between long-tail knowledge points and complex process flows.

[0028] Enhance data diversity and relevance: Generate three types of instruction question-and-answer samples: atomic, aggregated, and multi-hop, to adapt to the model's training needs for mastering basic knowledge points, integrating complex information, and multi-step reasoning ability; through the "understanding loss" mechanism, accurately focus on the model's weak knowledge points and improve training efficiency.

[0029] Ensuring data timeliness and maintainability: Achieving incremental automatic updates to the knowledge graph and instruction dataset, enabling timely incorporation of new standards, processes, and materials related to shipbuilding, and preventing training data from becoming outdated.

[0030] Promoting the professional application of large language models in the shipbuilding industry: After fine-tuning the data generated by this method, the reliability of the model's professional question-and-answer function and its ability to explain complex scenarios are significantly improved, making it more in line with the actual engineering needs of the shipbuilding industry. Attached Figure Description

[0031] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the following description of the embodiments will be briefly introduced. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0032] Figure 1 This is a schematic diagram of the overall architecture of the knowledge graph-based method for generating high-quality instruction data in the field of shipbuilding involved in this invention.

[0033] Figure 2 This is a schematic diagram of the relationship understanding assessment and confidence / loss calculation process involved in this invention.

[0034] Figure 3 This is a schematic diagram of the K-hop subgraph extraction and graph organization process involved in this invention.

[0035] Figure 4 This is a schematic diagram of the three types of QA (atomic / polymer / multi-hop) involved in this invention. Detailed Implementation

[0036] In the following description, only certain exemplary embodiments are briefly described. As those skilled in the art will recognize, the described embodiments can be modified in various ways without departing from the spirit or scope of the embodiments of the invention. Therefore, the drawings and description are considered to be exemplary in nature and not restrictive.

[0037] The following disclosure provides many different implementations or examples for carrying out different structures of the embodiments of the present invention. To simplify the disclosure of the embodiments of the present invention, specific examples of components and arrangements are described below. Of course, these are merely examples and are not intended to limit the embodiments of the present invention. Furthermore, reference numerals and / or reference letters may be repeated in different examples of the embodiments of the present invention; such repetition is for simplification and clarity and does not in itself indicate a relationship between the various implementations and / or arrangements discussed.

[0038] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0039] This embodiment provides a method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs. This method provides high-quality instruction data for supervised fine-tuning training of large-scale language models in a specific domain. The method includes:

[0040] (1) Collect multi-source heterogeneous text data in a specific field, preprocess the multi-source heterogeneous text data, and obtain standardized text corpus;

[0041] (2) Automatically extract key domain entities and their semantic relationships from the standardized text corpus, merge duplicate entities and relationships, resolve conflicting information using confidence level or manual verification, and construct an extensible domain knowledge graph; the domain knowledge graph consists of a set of key entities and a set of relationships;

[0042] (3) Use the training model to evaluate the understanding of each knowledge point in the domain knowledge graph and calculate the understanding loss of each knowledge point; mark the weak knowledge points and the mastered knowledge points according to the understanding loss threshold;

[0043] (4) Using the relationships in the domain knowledge graph as starting edges, and based on the understanding loss, depth constraint and length constraint, a specified traversal strategy is used to extract the subgraph to obtain the target subgraph;

[0044] (5) Based on the structural features of the target subgraph, automatically generate three types of instruction-response pairs: atomic, aggregate, and multi-hop.

[0045] (6) Monitor the updates and changes of the data source, repeat the preprocessing and knowledge extraction process of steps (1)-(2) for the newly added or modified text data, and incorporate the incremental knowledge into the domain knowledge graph to form a differential list containing new relationships, modified relationships and invalid relationships.

[0046] (7) For the new and modified relationships in the difference list, repeat steps (3)-(5) to generate new instruction-response pairs, mark or delete the original instruction-response pairs corresponding to the failed relationships, and conduct quality audits on the new instruction-response pairs;

[0047] (8) Perform multi-dimensional quality assessment on all instruction-response pairs, filter or regenerate substandard samples, and finally output a qualified high-quality supervised fine-tuning training dataset.

[0048] In some embodiments, the multi-source heterogeneous text data specifically includes enterprise internal documents, industry standards and specifications, published papers, patent documents, professional evaluation standards, design specifications, and process procedures in a specific field; the preprocessing specifically includes: performing format parsing and plain text extraction on the multi-source heterogeneous text data, removing noise such as headers and footers, duplicate page numbers, and invalid symbols, segmenting sentences or paragraphs, and performing terminology standardization processing on synonyms and abbreviations.

[0049] In some embodiments, the key entities in the domain mentioned in step (2) include component classes, process flow classes, material classes and standard specification classes; the semantic relationships include component-function, component-composition, process-step, process-materials used, process-standards followed, component-applicable specifications; the construction of the domain knowledge graph also includes introducing a domain ontology model, mapping the extracted entities to predefined categories, limiting the available relational patterns, and then writing the entities and relationships into a graph database to achieve unified storage and query.

[0050] In some embodiments, the specific process of understanding evaluation in step (3) is as follows: each relation in the domain knowledge graph is converted into a natural language statement, multiple restatements and negations of the statement are generated using a synthesizer model, yes / no judgment prompts are constructed and input into the training model, the judgment confidence of the training model is obtained, the relation confidence and understanding loss of the relation are calculated based on the confidence, and when the understanding loss is higher than a preset threshold, it is marked as a weak knowledge point; otherwise, it is marked as a mastered knowledge point.

[0051] In some embodiments, the depth constraint in step (4) is to limit the maximum number of hops in the subgraph extraction, and the length constraint is to limit the upper limit of the premise text length corresponding to the subgraph; the specified traversal strategy includes three types: a strategy that prioritizes the selection of edges with high understanding loss, a strategy that prioritizes the selection of edges with low understanding loss, and a strategy that selects edges uniformly and randomly, and a single strategy or a combination strategy is selected according to the training requirements.

[0052] In some embodiments, the atomic instruction-response pair in step (5) is generated based on a single-sided or single-node subgraph to construct a question and answer for a single knowledge point; the aggregated instruction-response pair is generated based on a subgraph containing multiple relationships within the same topic domain, and after restating, compressing and summarizing the relationships through a synthesizer model to form a natural language answer, the corresponding question is generated in reverse; the multi-hop instruction-response pair is generated based on a subgraph spanning multiple relationships to construct a question requiring multi-step reasoning, and the answer depends on the combination of multiple knowledge chains.

[0053] In some embodiments, the updates to the data source in step (6) specifically include text updates corresponding to new specifications in a specific domain, modifications to processes, and introduction of new materials; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed parts are supplemented or corrected; the quality audit and the multi-dimensional quality assessment in step (8) adopt the same standard.

[0054] In some embodiments, the multi-dimensional quality assessment in step (8) includes a language dimension, a knowledge dimension, and a structural dimension. The language dimension assesses grammatical correctness, terminology standardization, and clarity of expression. The knowledge dimension assesses the consistency between the answer and the facts in the knowledge graph and the absence of illusions. The structural dimension assesses the distribution of question types, the distribution of difficulty levels, and the repetition rate. The quality assessment also includes a comprehensive evaluation, which combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on closed-book knowledge-intensive tasks.

[0055] In some embodiments, when generating the instruction-response pairs, language diversity is enhanced by: designing multiple template questions for each type of semantic relationship, rewriting the template questions in multiple rounds using a large language model to form multiple questions that are semantically equivalent but express different meanings, and merging multiple knowledge points into the same question in aggregated and multi-hop instruction-response pairs to increase the difficulty of comprehensive understanding.

[0056] In some embodiments, the specific domain is the shipbuilding domain, the domain knowledge graph constructed in step (2) includes the welding process-step-materials-inspection standard subdomain, and the instruction-response pairs generated in step (5) involve atomic, aggregate and multi-hop instruction-response pairs related to welding method operation steps, weld quality judgment standards and defect handling measures.

[0057] The technical concept of this invention is as follows:

[0058] This invention revolves around a technical solution for "automatic synthesis of SFT instruction data in the shipbuilding field driven by knowledge graphs." The core process is "multi-source data acquisition—domain knowledge graph construction—understanding, evaluation, and graph organization—instruction sample generation—automatic updating and quality assessment."

[0059] First, unstructured / semi-structured texts from multiple sources, including internal documents of shipbuilding companies, industry standards, and publicly available papers and patents, are collected and preprocessed to form a standardized corpus.

[0060] Based on natural language processing technology and pre-trained large models, key entities (components, processes, materials, standards and specifications, etc.) and semantic relationships (component-function, process-step, process-standard, etc.) are extracted from the corpus to construct an scalable knowledge graph for the shipbuilding domain.

[0061] The model is trained to score the "understanding" of each knowledge point in the knowledge graph, and the understanding loss is calculated. The target subgraph is then selected by combining the K-hop subgraph extraction algorithm with various traversal strategies.

[0062] Based on the subgraph structure, three types of instruction-response pairs are automatically generated: atoms (single knowledge point Q&A), aggregates (summary / comparison Q&A of multiple related knowledge points), and multi-hops (multi-step reasoning Q&A). The language diversity is improved by rewriting templates.

[0063] An automatic update mechanism is established to monitor changes in data sources and incrementally update the knowledge graph and dataset. At the same time, the synthetic data is subjected to dual quality assessment through multi-dimensional indicators of language, knowledge, and structure, as well as the performance of downstream models, to ensure that the data meets the standards.

[0064] The purpose of this invention is to propose a high-quality supervised fine-tuning (SFT) data construction method for the shipbuilding industry. This method can make full use of knowledge graphs to perform structured modeling of domain knowledge and automatically generate multi-type (atomic, aggregate, multi-hop) instruction question-and-answer samples through templated and diversified strategies to solve the problems of high cost, high illusion rate and low coverage of traditional data construction.

[0065] The specific objectives of this invention include:

[0066] This provides a mechanism for extracting expertise from multi-source unstructured text and constructing scalable knowledge graphs;

[0067] A mechanism is provided to automatically generate diverse, high-quality instruction-response pairs based on graph structure for fine-tuning language models;

[0068] We provide a graph traversal and subgraph organization strategy that combines the understanding of entity relationships with priority coverage of weak knowledge points;

[0069] An automatic update mechanism for instruction datasets is provided to ensure that model training data is dynamically synchronized with knowledge evolution;

[0070] We provide a complete data quality assessment system to ensure that the generated content meets high standards in terms of accuracy, expressiveness, and consistency.

[0071] By achieving the above objectives, this invention can significantly reduce the cost of manual annotation, improve the coverage, reliability, and timeliness of model training data, and provide strong support for the professional application of large language models in the industrial field.

[0072] like Figure 1 As shown, the synthesis algorithm proposed in this invention Using a raw corpus from the shipbuilding field as input, a structured synthetic instruction dataset is obtained through a three-stage process of "extraction-organization-generation". Its formal representation is as follows:

[0073] ;

[0074] Original texts in the shipbuilding field from multiple sources, including internal company documents, industry standards, published papers, and patents.

[0075] : Represents "knowledge extraction operation", which is from The process of extracting information and constructing a knowledge graph;

[0076] KG=(E,R): A knowledge graph of the shipbuilding domain consisting of a set of key entities E and a set of relations R;

[0077] It refers to "data organization operations," which is the process of transforming structured knowledge into instruction-response pairs based on KG.

[0078] The final high-quality SFT training data generated is in the form of prompt-response.

[0079] The method consists of the following main functions:

[0080] 1. Data Sources and Acquisition: Responsible for acquiring and preprocessing multi-source text data;

[0081] 2. Domain Knowledge Graph Construction: Responsible for entity recognition, relation extraction, and ontology modeling to form a scalable KG;

[0082] 3. Understanding and Evaluation: Through training the model Each knowledge point is scored based on its "comprehension level";

[0083] 4. Subgraph Extraction and Graph Organization: K-hop subgraph extraction based on loss constraints and length constraints;

[0084] 5. Instruction Generation: Generates atomic / aggregate / multi-hop QA based on entity-relationship schemas and subgraph structures;

[0085] 6. Automatic Updates: Monitors changes in the data source and performs incremental updates to the KG and QA sets;

[0086] 7. Quality Assessment: Utilizing multidimensional indicators to assess... Conduct a joint assessment of intrinsic quality and downstream effects.

[0087] The comprehensive quality index Q can be formalized as:

[0088] ;

[0089] in To measure the quality of the data itself, Measuring the model after SFT Performance on closed-book, knowledge-intensive tasks; Metric: refers to a set of multi-dimensional quality assessment indicators (a set of predefined indicators), including specific quantitative indicators for language dimension (grammatical correctness, terminology standardization, clarity of expression), knowledge dimension (factual consistency, no illusions), and structural dimension (distribution of question types, distribution of difficulty levels, repetition rate); : Refers to the test dataset (i.e., the evaluation dataset) used to evaluate the model's performance; : refers to the target large language model (i.e., the trained model) after being fine-tuned by Dsynth generated by this invention.

[0090] ;

[0091] Where L is the language dimension score (range 0-100), which is calculated by weighting grammatical correctness (weight 0.3), terminology standardization (weight 0.4), and clarity of expression (weight 0.3);

[0092] K is the knowledge dimension score (range 0-100), which is calculated by weighting factual consistency (weight 0.7) and the rate of no illusion (weight 0.3);

[0093] S is the structural dimension score (range 0-100), which is calculated by weighting the reasonableness of the problem type distribution (weight 0.3), the uniformity of the difficulty level distribution (weight 0.3), and the sample duplication rate (reverse scoring, weight 0.4).

[0094] For the weights of each dimension, satisfy The default values ​​are 0.3, 0.5, and 0.2, which can be adjusted according to the needs of the field.

[0095] ;

[0096] Where P represents precision, R represents recall, and F1 represents the F1 score, all based on the model's performance. The accuracy rate of the questions and answers on the platform was statistically derived (value range 0-1).

[0097] α, β, and γ are the indicator weights, satisfying α+β+γ=1. The default value is 1 / 3 for each of them, which can be adjusted according to the task focus (such as prioritizing the accuracy or completeness of the answer).

[0098] Functional structure and working principle:

[0099] Data Sources and Collection:

[0100] Data sources include:

[0101] Internal documents of shipbuilding companies: such as hull structure design specifications, compartment layout specifications, cable laying procedures, welding / painting process cards, etc.

[0102] Industry standards and specifications: such as classification society specifications, welding procedure qualification standards, material selection standards, etc.;

[0103] Publicly published papers and patents: such as research results on ship structure optimization, process automation, and intelligent inspection.

[0104] The aforementioned unstructured / semi-structured text will be preprocessed uniformly, including:

[0105] Format parsing and plain text extraction;

[0106] Remove noise such as headers and footers, duplicate page numbers, and invalid symbols;

[0107] Sentence / paragraph segmentation and terminology standardization (synonyms and abbreviations are unified).

[0108] The obtained standardized text corpus is used as input for subsequent knowledge extraction.

[0109] Domain knowledge graph construction:

[0110] Using preprocessed corpus as input, key entities and their relationships in the shipbuilding domain are automatically extracted to construct a domain key-value group (KG). Key entities include, but are not limited to:

[0111] Components: hull, hatch covers, main engine, auxiliary engine, welded parts, support components, etc.

[0112] Process flow: material preparation, cutting, panel assembly, 3D assembly, welding, flaw detection, etc.

[0113] Materials: Various types of marine steel, alloy materials, welding materials, coatings, etc.

[0114] Standards and specifications: such as a classification society's specifications, process evaluation standard numbers, etc.

[0115] Relationships represent semantic connections between entities, for example:

[0116] Component – ​​Function;

[0117] Components – Composition;

[0118] Process – steps;

[0119] Process – Materials used;

[0120] Process – Follows standards;

[0121] Components – applicable specifications, etc.

[0122] In terms of implementation, it combines natural language processing techniques with pre-trained large models:

[0123] 1. Using LLM Parse paragraph text into knowledge statements in the form of triples;

[0124] 2. Introduce a domain ontology model to map entities to predefined categories and limit the available relational patterns (such as "component-function", "process-step", etc.).

[0125] 3. Merge entities and relationships that appear repeatedly across segments, and resolve conflicting information using confidence level or manual verification;

[0126] 4. Write the obtained entities and relationships into a graph database to achieve unified storage and querying.

[0127] In this way, the knowledge that was originally scattered and distributed in natural language in long documents is converged into a structured graphical representation, which significantly reduces the probability of illusion in the subsequent generation process and provides a foundation for automatic reasoning.

[0128] Understanding assessment and relationship confidence calculation (corresponding) Figure 2 ):

[0129] To make the generated data more targeted, this invention proposes to train the model. The level of understanding at each knowledge point is explicitly measured. For each relationship in the diagram... Its natural language description can be regarded as a statement Theoretically, the true probability is 1:

[0130] ;

[0131] use for Generate a set of restatements and its negation form and through prompts Given the confidence level of a "yes / no" judgment, we define:

[0132] ;

[0133] :relation Confidence level; n: the relationship involved The total number of samples; : A relationship exists in the j-th sample. "the situation"; There is no relation in the j-th sample. "the situation"; : A relationship exists in sample j The probability of "correct match (t represents true)" at that time; There is no relation in sample j. The probability of a "mismatch (f represents false)" at that time;

[0134] And define the loss of understanding:

[0135] ;

[0136] :relation The corresponding understanding loss (quantifying the model's error in understanding the relationship);

[0137] Understanding loss characterizes The gap between "completely unaware" and "completely mastered" of this knowledge point provides a quantitative basis for subsequent subgraph extraction and training data selection: the higher the loss, the weaker the knowledge point is in the current model, and relevant training samples should be generated first.

[0138] Subgraph extraction and graph organization (corresponding) Figure 3 ):

[0139] In the preferred embodiment, the subgraph is the basis for generating the smallest QA unit. This invention employs the K-hop subgraph extraction algorithm, starting from a primary relation... Starting from the beginning, perform a constrained traversal of the graph; : refers to the "source entity", which is the starting entity of the relation instance (such as "hull section" in the shipbuilding field).

[0140] : Refers to the "target entity," which is the target entity of this relation instance (e.g., "welding process" corresponding to "hull section"). The pseudocode is as follows:

[0141] Input: Graph G, initial edge R_i, traversal strategy S;

[0142] Output: Subgraph G′;

[0143] 1. G′←{R_i};

[0144] 2. Candidate set C ← Adjacent edge set ( );

[0145] 3. while C ≠ ∅ do;

[0146] a. Select edge e according to S;

[0147] bG′←G′∪{e};

[0148] c. Terminate if the constraints are satisfied;

[0149] d. Add the adjacent edges of the two nodes of e to C;

[0150] Return G′;

[0151] Constraints include two types:

[0152] Depth constraint: Limits the maximum number of hops K, controlling the length of the inference chain;

[0153] Length constraint: Set an upper limit on the pre_length of the prerequisite token for the subgraph to prevent the subgraph from becoming too large and causing an imbalance in the sample distribution.

[0154] Traversal strategy S supports the following options:

[0155] 1. max_loss: Prioritize edges with high understanding loss, focusing on generating knowledge that the model "does not" know;

[0156] 2. min_loss: Prioritize edges with low understanding loss to construct stable links as high-quality context;

[0157] 3. Random: Uniformly and randomly selected to ensure coverage.

[0158] By combining the above strategies, a balanced control over the complexity, relevance, and computational cost of subgraphs can be achieved.

[0159] Instruction generation mechanism (corresponding) Figure 4 ):

[0160] Based on the constructed knowledge graph and extracted subgraphs, this invention designs a mechanism for automatically generating prompt-response pairs.

[0161] 1. Atomic QA

[0162] When the subgraph has a single edge or a single node, it generates question-and-answer responses for a single knowledge point. For example, triples.

[0163] <Component X, Function Y>=>

[0164] Q: "What is the function of component X?"

[0165] A: "The function of component X is Y."

[0166] 2. Aggregated QA

[0167] When a subgraph contains multiple interrelated relationships belonging to the same subject area (e.g., multiple steps of a process, multiple components of a device), this invention utilizes a synthesizer model. These relationships are "restated, compressed, and summarized" to generate a complete explanatory natural language answer, which is then used to automatically generate the corresponding question.

[0168] This type of QA is used to examine the model's summarization ability, comparison ability, and structured information integration ability, and is applicable to scenarios such as process flows, specification clauses, component compositions, etc. For example, the triples are as follows:

[0169] <GMAW welding process, step, root cleaning>;

[0170] <GMAW welding process, step, hot welding>;

[0171] <GMAW welding process, step, filling>;

[0172] <Root cleaning, tool used, angle grinder>;

[0173] Automatically generated restated answer ( Output) is as follows:

[0174] The GMAW gas shielded welding process usually includes three main steps: root cleaning, hot welding, and filling.

[0175] Among them, the root cleaning step requires using an angle grinder to grind the root of the weld.

[0176] The automatically generated questions are as follows:

[0177] What steps does the GMAW gas shielded welding process include? What tool is required for the root cleaning step?

[0178] The formed aggregated QA is as follows:

[0179] Question: What steps does the GMAW welding process include? What tools are required for the root cleaning step?

[0180] Answer: This process includes three steps: root cleaning, hot welding, and filling. Among them, the root cleaning step requires using an angle grinder.

[0181] 3. Multi-hop QA

[0182] For subgraphs spanning multiple relationships, explicitly construct questions that require multi-step reasoning, making the answer depend on the combination of multiple knowledge chains, thereby training the model's reasoning ability in scenarios such as complex processes, system compositions, and causal relationships. For example, the triples are as follows:

[0183] <Bulkhead panel model X12, material, AH36 steel>;

[0184] <AH36 steel, applicable welding process, CO2 gas shielded welding>;

[0185] <CO2 gas shielded welding, inspection standard, classification society specification 5.4.2>; ​​​​Automatically generated restatement of answers ( The output is as follows:

[0188] The bulkhead plate X12 is made of AH36 high-strength marine steel, so the applicable welding process is CO2 gas shielded welding.

[0189] According to industry standards, CO2 gas shielded welding should comply with Clause 5.4.2 of the classification society's regulations.

[0190] The automatically generated questions are as follows:

[0191] What welding process should be used for the X12 bulkhead panel, and what inspection standards must it meet?

[0192] The resulting multi-hop QA is as follows:

[0193] Q: What welding process should be used for the X12 bulkhead? What inspection standards should this process follow?

[0194] A: The applicable welding process for bulkhead X12 is CO2 gas shielded welding, which should comply with the classification society's standard 5.4.2.

[0195] To enhance linguistic diversity, this invention:

[0196] Design multiple template questions for each type of relationship;

[0197] LLM is used to rewrite the template in multiple rounds to form multiple questions that are semantically equivalent but express different meanings;

[0198] In combined question-and-answer sessions, multiple knowledge points are merged into the same question, increasing the difficulty of comprehensive comprehension.

[0199] During the generation process, the system records the source triple ID or subgraph ID for each question-and-answer sample to support subsequent updates and traceability.

[0200] Automatic updates and maintenance:

[0201] When the data source changes (new specifications, modified processes, introduction of new materials, etc.), the automatic update of this invention is responsible for:

[0202] 1. Repeat the aforementioned "preprocessing + knowledge extraction" process for new data, and incorporate the incremental knowledge into the KG;

[0203] 2. Compare the knowledge sets before and after the update to form a difference list (newly added relationships, modified relationships, and invalid relationships);

[0204] 3. Generate new QA samples for newly added / modified relationships using the command generation function, and mark or delete samples related to invalid relationships;

[0205] 4. Trigger the quality assessment process to review the incremental samples.

[0206] Thanks to the binding design of question-answer samples and triples, the entire update process can be completed within the incremental range without rebuilding the entire dataset, thus ensuring real-time synchronization between training data and domain knowledge.

[0207] Synthetic data quality assessment:

[0208] right Multi-dimensional scoring is conducted, including but not limited to:

[0209] Language dimension: grammatical correctness, terminology standardization, and clarity of expression;

[0210] Knowledge dimension: Does the answer strictly align with the facts in KG, and is there any illusion involved?

[0211] Structural dimensions: distribution of problem types, distribution of difficulty levels, repetition rate, etc.

[0212] And a comprehensive evaluation is conducted using the following formula:

[0213] ;

[0214] The metric is a set of predefined metrics. Samples that do not meet the metrics will be filtered or regenerated to ensure that all data entering the SFT training are of high quality.

[0215] Best Practice Implementation: SFT Data Construction for Ship Welding Procedure Specifications

[0216] In a specific and preferred embodiment, the present invention is applied to the training of a "hull welding process specification auxiliary question-and-answer model":

[0217] 1. Collect texts from a shipyard's welding process specifications, weld quality inspection standards, welding material selection manual, and relevant standard clauses;

[0218] 2. Construct a knowledge graph subdomain containing "welding process → steps → materials used → inspection standards";

[0219] 3. Based on this subdomain, a large number of atomic, aggregated, and multi-hop questions and answers are generated regarding "operational steps of a certain welding method", "weld quality judgment criteria", and "defect handling measures";

[0220] 4. After quality screening of the generated data, it is used to train the general Chinese large-scale model using SFT;

[0221] 5. After training, the model can accurately answer welding process-related questions under closed-book conditions and can provide multi-step explanations for complex scenarios, such as "how to handle incomplete penetration defects in longitudinal butt welds of ship hulls according to specifications".

[0222] Experiments show that after fine-tuning using the data generated by this invention, the model's accuracy and stability in professional question answering in welding scenarios are significantly better than the baseline method that only uses "a small number of manual examples + general synthesis".

[0223] Comparison and improvements with the background technology:

[0224] Compared with existing methods in the background art, the present invention has significant improvements in the following aspects:

[0225] 1. From "Text-Driven" to "Graph-Driven" Data Synthesis Paradigm

[0226] Traditional methods often rely directly on raw documents or fragmented paragraphs to generate questions and answers using LLM (Local Management Model), lacking explicit modeling of the knowledge structure. This can easily lead to uneven knowledge coverage, omissions of long-tail information, and factual illusions. This invention, by constructing a domain knowledge graph, explicitly organizes knowledge points in a graph structure and uses this as the "skeleton" for data synthesis, significantly improving the systematicity and accuracy of knowledge coverage.

[0227] 2. Introduce a data selection mechanism guided by "understanding loss".

[0228] Existing methods typically do not distinguish between knowledge points already mastered and those not mastered by the model, or select training samples based solely on a rough difficulty estimate. This invention defines... Equal measurement, explicit measurement By assessing the level of understanding of each knowledge point and combining it with the K-hop graph organization strategy, more training samples can be generated specifically for the model's "weak knowledge" areas, thereby improving training efficiency and effectiveness.

[0229] 3. A unified framework supporting atomic, aggregated, multi-hop, and multi-level QA.

[0230] Traditional synthesis methods often only cover basic question answering and lack systematic support for multi-relational reasoning and complex scenarios. This invention, based on subgraph size and structure, uniformly generates atomic QA, aggregated QA, and multi-hop QA, ensuring comprehensive coverage of basic knowledge while systematically enhancing the model's multi-step reasoning capabilities, making the fine-tuned model more closely resemble real-world engineering problems.

[0231] 4. Automatic update and traceable data maintenance mechanism

[0232] In existing technologies, knowledge updates often require manual reconstruction of datasets, which is costly and prone to errors. This invention achieves incremental updates and tracking of synthetic data through a three-layer binding mechanism of "sample-triple-subgraph" combined with a differential detection mechanism. With the introduction of new standards and processes, the model can continuously absorb new knowledge through multiple fine-tuning processes, avoiding the emergence of "outdated answers."

[0233] 5. A dual evaluation system combining intrinsic quality and downstream performance.

[0234] Traditional methods often rely on manual spot-checks or a small amount of automated metrics to evaluate synthetic data, making it difficult to systematically measure its true value to downstream tasks.

[0235] This invention is achieved through ;

[0236] By incorporating both the quality of the data itself and the performance of the fine-tuned model on closed-book, knowledge-intensive tasks into a unified metric, we can ensure that "good data" truly leads to "good models."

[0237] In summary, this invention has made systematic improvements to the existing technology in terms of overall architecture, methodology, and evaluation system, realizing the automatic construction of high-quality and sustainable SFT instruction data for the shipbuilding industry, and has significant engineering practical value and promotion prospects.

[0238] The embodiments described above are for illustrative purposes only and are not intended to limit the invention. Therefore, any changes in numerical values ​​or substitutions of equivalent elements should still fall within the scope of this invention.

[0239] The above detailed description will enable those skilled in the art to understand that the present invention can indeed achieve the aforementioned objectives and has complied with the provisions of the Patent Law.

[0240] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the invention. The above descriptions are merely preferred embodiments of the invention and are not intended to limit the invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the invention should be included within the scope of protection of the invention.

[0241] It should be noted that the above description of the process is for illustrative purposes only and does not limit the scope of this specification. Those skilled in the art can make various modifications and changes to the process under the guidance of this specification. However, these modifications and changes remain within the scope of this specification.

[0242] The basic concepts have been described above. Obviously, for those skilled in the art who have read this application, the above disclosure is merely illustrative and does not constitute a limitation of this application. Although not explicitly stated herein, those skilled in the art may make various modifications, improvements, and corrections to this application. Such modifications, improvements, and corrections are suggested in this application, and therefore, such modifications, improvements, and corrections still fall within the spirit and scope of the exemplary embodiments of this application.

[0243] Furthermore, this application uses specific terms to describe its embodiments. For example, "an embodiment," "one embodiment," and / or "some embodiments" refer to a particular feature, structure, or characteristic related to at least one embodiment of this application. Therefore, it should be emphasized and noted that "an embodiment," "one embodiment," or "an alternative embodiment" mentioned twice or more in different positions in this specification do not necessarily refer to the same embodiment. In addition, certain features, structures, or characteristics in one or more embodiments of this application can be appropriately combined.

[0244] Furthermore, those skilled in the art will understand that aspects of this application can be described and illustrated through several patentable types or situations, including any new and useful combination of processes, machines, products, or substances, or any new and useful improvements thereof. Therefore, aspects of this application can be implemented entirely in hardware, entirely in software (including firmware, resident software, microcode, etc.), or a combination of hardware and software. All of the above hardware or software can be referred to as a “unit,” “module,” or “system.” Furthermore, aspects of this application can take the form of a computer program product embodied in one or more computer-readable media, wherein computer-readable program code is contained therein.

[0245] The computer program code required for the operation of each part of this application can be written in any one or more programming languages, including object-oriented programming languages ​​such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, and Python; general programming languages ​​such as C; Visual Basic, Fortran2103, Perl, COBOL2102, PHP, and ABAP; dynamic programming languages ​​such as Python, Ruby, and Groovy; or other programming languages. This program code can run entirely on the user's computer, or as a standalone software package on the user's computer, or partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the latter case, the remote computer can be connected to the user's computer via any network, such as a local area network (LAN) or wide area network (WAN), or connected to an external computer (e.g., via the Internet), or in a cloud computing environment, or used as a service such as Software as a Service (SaaS).

[0246] Furthermore, unless expressly stated in the claims, the order of processing elements and sequences, the use of numbers and letters, or other names described in this application are not intended to limit the order of the processes and methods of this application. Although some currently considered useful embodiments of the invention have been discussed in the foregoing disclosure by way of various examples, it should be understood that such details are for illustrative purposes only, and the appended claims are not limited to the disclosed embodiments; rather, the claims are intended to cover all modifications and equivalent combinations that conform to the substance and scope of the embodiments of this application. For example, although the implementation of the various components described above can be embodied in a hardware device, it can also be implemented as a purely software solution, such as an installation on an existing server or mobile device.

[0247] Similarly, it should be noted that, in order to simplify the description of the present application and thus aid in the understanding of one or more embodiments of the invention, the foregoing description of the embodiments of the present application sometimes combines multiple features into a single embodiment, drawing, or description thereof. However, this approach of the present application should not be construed as reflecting an intention that the claimed subject matter requires more features than expressly recited in each claim. Rather, the subject of the invention should possess fewer features than in any single embodiment described above.

Claims

1. A method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs, characterized in that, include: (1) Collect multi-source heterogeneous text data in a specific field, preprocess the multi-source heterogeneous text data, and obtain standardized text corpus; (2) Automatically extract key domain entities and their semantic relationships from the standardized text corpus, merge duplicate entities and relationships, resolve conflicting information using confidence level or manual verification, and construct an scalable domain knowledge graph. (3) Use the training model to evaluate the understanding of each knowledge point in the domain knowledge graph and calculate the understanding loss of each knowledge point. The specific process of the understanding evaluation is as follows: convert each relation in the domain knowledge graph into a natural language statement, use the synthesizer model to generate multiple restatements and negations of the statement, construct yes / no judgment prompts and input them into the training model, obtain the judgment confidence of the training model output, calculate the relation confidence and understanding loss of the relation based on the confidence, and mark the understanding loss as a weak knowledge point when it is higher than a preset threshold, otherwise mark it as a mastered knowledge point. (4) Using the relationships in the domain knowledge graph as starting edges, and based on the understanding loss, depth constraint and length constraint, a specified traversal strategy is used to extract the subgraph to obtain the target subgraph; (5) Based on the structural features of the target subgraph, automatically generate three types of instruction-response pairs: atomic, aggregate, and multi-hop. (6) Monitor the updates and changes of the data source, repeat the preprocessing and knowledge extraction process of steps (1)-(2) for the newly added or modified text data, and incorporate the incremental knowledge into the domain knowledge graph to form a differential list containing new relationships, modified relationships and invalid relationships. (7) For the newly added and modified relations in the difference list, repeat steps (3)-(5) to generate new instruction-response pairs; (8) Perform multi-dimensional quality assessment on all instruction-response pairs, filter or regenerate substandard samples, and finally output a qualified high-quality supervised fine-tuning training dataset.

2. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The multi-source heterogeneous text data specifically includes internal enterprise documents, industry standards and specifications, published papers, patent documents, professional evaluation standards, design specifications, and process procedures in specific fields; the preprocessing specifically includes: format parsing and plain text extraction of the multi-source heterogeneous text data, removing noise such as headers and footers, duplicate page numbers, and invalid symbols, segmenting sentences or paragraphs, and standardizing the terminology of synonyms and abbreviations.

3. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The key entities in the domain mentioned in step (2) include component class, process flow class, material class and standard specification class; the semantic relationships include component-function, component-composition, process-step, process-materials used, process-standards followed, component-applicable specifications; the construction of the domain knowledge graph also includes introducing a domain ontology model, mapping the extracted entities to predefined categories, limiting the available relational patterns, and then writing the entities and relations into a graph database to achieve unified storage and query.

4. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The depth constraint in step (4) is to limit the maximum number of hops in the subgraph extraction, and the length constraint is to limit the upper limit of the premise text length corresponding to the subgraph. The specified traversal strategy includes three types: the strategy of prioritizing the selection of edges with high understanding loss, the strategy of prioritizing the selection of edges with low understanding loss, and the strategy of uniformly and randomly selecting edges. A single strategy or a combination of strategies can be selected according to the training requirements.

5. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The atomic instruction-response pairs described in step (5) are generated based on a single-sided or single-node subgraph and construct a question-and-answer pair for a single knowledge point; the aggregated instruction-response pairs are generated based on a subgraph containing multiple relationships within the same topic domain, and after restating, compressing and summarizing the relationships through a synthesizer model to form a natural language answer, the corresponding question is generated in reverse; the multi-hop instruction-response pairs are generated based on a subgraph spanning multiple relationships, constructing a question that requires multi-step reasoning, and the answer depends on the combination of multiple knowledge chains.

6. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The specific changes to the data source in step (6) include text updates corresponding to new specifications, modified processes, and the introduction of new materials in a specific field; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed parts are supplemented or corrected. The quality audit and the multi-dimensional quality assessment in step (8) adopt the same standard.

7. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The multi-dimensional quality assessment in step (8) includes language dimension, knowledge dimension and structural dimension. The language dimension assesses grammatical correctness, terminology standardization and clarity of expression. The knowledge dimension assesses the consistency between the answer and the facts in the knowledge graph and the absence of illusions. The structural dimension assesses the distribution of question types, the distribution of difficulty levels and the repetition rate. The quality assessment also includes a comprehensive evaluation, which combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on closed-book knowledge-intensive tasks.

8. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 5, characterized in that, When generating the instruction-response pairs, language diversity is enhanced by designing multiple template questions for each type of semantic relationship, rewriting the template questions in multiple rounds using a large language model to form multiple questions that are semantically equivalent but express different meanings, and merging multiple knowledge points into the same question in aggregated and multi-hop instruction-response pairs to increase the difficulty of comprehensive understanding.

9. The method for generating high-quality instruction data in the shipbuilding field based on knowledge graphs according to claim 1, characterized in that, The specific domain is the shipbuilding domain. The domain knowledge graph constructed in step (2) includes the welding process-step-materials-inspection standard subdomain. The instruction-response pairs generated in step (5) involve atomic, aggregate and multi-hop instruction-response pairs related to welding method operation steps, weld quality judgment standards and defect handling measures.

Images