Method for generating high-quality instruction data in ship manufacturing field based on knowledge graph

Abstract

The invention provides a method for generating high-quality instruction data in the ship manufacturing field based on a knowledge graph. The method comprises the following steps: (1) collecting and preprocessing a multi-source text; (2) knowledge extraction and graph construction; (3) knowledge point understanding evaluation and marking; (4) extracting constrained sub-graphs; (5) generating a multi-type instruction-response pair; (6) data source monitoring and map increment updating; (7) the instruction-response pair increment is updated and audited; and (8) performing multi-dimensional quality evaluation and output on the instruction data. The invention aims to solve the problems of high training data construction cost, high illusion rate, low coverage and poor timeliness in a large language model fine tuning process in the ship manufacturing field.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of artificial intelligence and the shipbuilding industry, and in particular to a method for generating high-quality instruction data in the field of shipbuilding based on a knowledge graph. BACKGROUND

[0002] Supervised fine-tuning (SFT) of a large language model (LLM) is a core approach for the LLM to master professional knowledge in a field. However, this process highly depends on large-scale and high-quality supervised training data, which needs to be manually annotated by field experts, and therefore has the problems of high construction cost and long cycle, and is difficult to adapt to the demand for rapid update of professional knowledge.

[0003] The existing method for automatically synthesizing training data by using an LLM has obvious defects in a knowledge-intensive field such as shipbuilding: the generated content is prone to factual errors (model hallucination), the coverage of long-tail knowledge is insufficient, and the multi-step correlation of a complex process flow cannot be accurately expressed, and the question-and-answer structure is templated and lacks diversity, which may limit the performance of the model after fine-tuning and make it difficult to meet the demand of a high-reliability scenario.

[0004] The professional knowledge in the field of shipbuilding is large in scale and scattered in sources (covering unstructured documents such as design documents, process procedures, industry standards, and research papers), and the knowledge system has a fragmented characteristic and is difficult to be systematically utilized. Although the existing technology can assist in constructing a knowledge graph in the field of shipbuilding, it lacks a mechanism for automatically generating instruction-based question-and-answer training data based on the knowledge graph, and the knowledge graph lacks dynamic expansion capability and cannot timely incorporate the latest knowledge in the industry. SUMMARY

[0005] The application provides a method for generating high-quality instruction data in the field of shipbuilding based on a knowledge graph, and aims to solve the problems of high construction cost, high hallucination rate, low coverage, and poor timeliness of training data in the fine-tuning process of a large language model in the field of shipbuilding.

[0006] To achieve the above-mentioned purpose, the application adopts the following technical solutions: The method for generating high-quality instruction data in the field of shipbuilding based on a knowledge graph is used to provide high-quality instruction data for supervised fine-tuning training of a large language model in a specific field, and the method comprises the following steps: (1) Collecting multi-source heterogeneous text data in a specific field, preprocessing the multi-source heterogeneous text data, and obtaining standardized text corpus; (2) Automatically extracting domain key entities and their semantic relationships from the standardized text corpus, merging repeated entities and relationships, solving conflict information by using confidence or manual verification, and constructing an extensible domain knowledge graph; the domain knowledge graph is composed of a key entity set and a relationship set; (3) using the trained model to evaluate the understanding of each knowledge point in the domain knowledge graph, and calculating the understanding loss of each knowledge point; according to the understanding loss threshold, marking the weak knowledge points and mastered knowledge points; (4) taking the relationship in the domain knowledge graph as the starting edge, based on the understanding loss, depth constraint and length constraint, using a specified traversal strategy to extract subgraphs to obtain target subgraphs; (5) based on the structural characteristics of the target subgraph, automatically generating three types of instruction-response pairs of atomic type, aggregation type and multi-hop type; (6) listening to the update changes of the data source, repeating the preprocessing and knowledge extraction processes of steps (1)-(2) for the newly added or modified text data, incorporating the incremental knowledge into the domain knowledge graph, and forming a difference list containing new relationships, modified relationships and invalid relationships; (7) for the new relationships and modified relationships in the difference list, repeat steps (3)-(5) to generate new instruction-response pairs, mark or delete the original instruction-response pairs corresponding to the invalid relationships, and perform quality audit on the new instruction-response pairs; (8) multi-dimensional quality evaluation is performed on all instruction-response pairs, and non-compliant samples are filtered or regenerated, and finally a qualified high-quality supervision fine-tuning training data set is output.

[0007] In the specification, the multi-source heterogeneous text data specifically includes enterprise internal documents, industry standard specifications, public papers, patent documents, professional assessment standards, design specifications and process procedures in a specific field; the preprocessing specifically includes: format analysis and pure text extraction on multi-source heterogeneous text data, removing header and footer, repeated page numbers, invalid symbol noise, sentence or paragraph segmentation, and implementing term standardization processing on synonyms and abbreviations.

[0008] In the specification, the domain key entities in step (2) include parts, process flow, materials and standard specifications; the semantic relationships include part-function, part- composition, process-step, process-used material, process-followed standard, and part-applicable specification; the construction of the domain knowledge graph also includes introducing a domain ontology model, mapping the extracted entities to predefined categories, and limiting available relationship patterns, and then writing entities and relationships into a graph database to realize unified storage and query.

[0009] In the specification, the specific process of the understanding evaluation in step (3) is: converting each relationship in the domain knowledge graph into a natural language statement, generating multiple restatement forms and their negative forms of the statement using a synthesizer model, constructing yes / no judgment prompts and inputting a training model, obtaining the judgment confidence output by the training model, calculating the relationship confidence and understanding loss of the relationship based on the confidence, marking as a weak knowledge point when the understanding loss is higher than a preset threshold, otherwise marking as a mastered knowledge point.

[0010] In the specification, the depth constraint in step (4) is to limit the maximum hop number of subgraph extraction, and the length constraint is to limit the upper limit of the premise text length corresponding to the subgraph; the specified traversal strategy includes three types: a strategy of preferentially selecting edges with high understanding loss, a strategy of preferentially selecting edges with low understanding loss, and a strategy of uniformly and randomly selecting edges, and a single strategy or a combined strategy is selected according to training requirements.

[0011] In the specification, the atomic instruction-response pair in step (5) is generated based on a single edge or single node subgraph, and a question and answer are constructed for a single knowledge point; the aggregated instruction-response pair is generated based on a subgraph containing multiple same topic domain associated relationships, and after the associated relationships are restated, compressed and summarized by a synthesizer model to form a natural language answer, the corresponding question is generated in reverse; the multi-hop instruction-response pair is generated based on a subgraph across multiple relationships, a question requiring multi-step reasoning is constructed, and the answer depends on the combination of multiple knowledge chains.

[0012] In the specification, the update and change of the data source in step (6) specifically includes text updates corresponding to new specifications, modified processes, and introduced new materials in a specific field; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed part is supplemented or corrected, and the same standard as the multi-dimensional quality evaluation in step (8) is used for the quality audit.

[0013] In the specification, the multi-dimensional quality evaluation in step (8) includes language dimension, knowledge dimension and structure dimension, wherein the language dimension evaluates grammar correctness, term standardization and expression clarity, the knowledge dimension evaluates the consistency and non-illusion of the answer with the facts in the knowledge graph, and the structure dimension evaluates the problem type distribution, difficulty level distribution and repetition rate; the quality evaluation also includes comprehensive evaluation, and the comprehensive evaluation combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on a closed-book knowledge-intensive task.

[0014] In the specification, when generating the instruction-response pair, language diversity is improved by designing multiple template questions for each type of semantic relationship, using a large language model to rewrite the template questions in multiple rounds, forming multiple questions that are semantically equivalent but express different ideas, and in the aggregate and multi-hop instruction-response pair, combining multiple knowledge points into the same question to improve comprehensive understanding difficulty.

[0015] In the specification, the specific field is the shipbuilding field, the domain knowledge graph constructed in step (2) contains the welding process-step-material used-inspection standard sub-domain, and the instruction-response pair generated in step (5) involves atomic, aggregate and multi-hop instruction-response pairs related to welding method operation steps, weld quality judgment standards and defect treatment measures.

[0016] In summary, the present application has at least the following beneficial effects: Significantly reduce the cost of manual annotation, without relying on a large number of experts to manually construct training data, and realize the automatic synthesis of high-quality SFT data.

[0017] Improve the core quality of training data: through knowledge graph structured modeling, significantly reduce the data illusion rate, improve the systematicness and integrity of knowledge coverage, and cover the association between long-tail knowledge points and complex process flows.

[0018] Enhance data diversity and relevance: generate atomic, aggregate, and multi-hop instruction question and answer samples to adapt to the training needs of the model in terms of basic knowledge points, complex information integration, and multi-step reasoning ability; through the "understanding loss" mechanism, precisely focus on weak knowledge points of the model, and improve training efficiency.

[0019] Ensure data timeliness and maintainability: realize incremental automatic updating of knowledge graph and instruction data set, and can timely incorporate new specifications, new technologies and new materials related knowledge in the field of shipbuilding, and avoid outdated training data.

[0020] Promote the professional landing of large language models in the field of shipbuilding: after the data generated by the method is fine-tuned, the professional question and answer reliability and complex scene explanation ability of the model are significantly improved, and it is more suitable for the actual engineering needs of the shipbuilding industry. BRIEF DESCRIPTION OF DRAWINGS

[0021] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed in the embodiment description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0022] Figure 1A schematic diagram of the overall architecture of the knowledge graph-based high-quality instruction data generation method in the shipbuilding field involved in the present application.

[0023] Figure 2 A schematic diagram of the relationship understanding evaluation and confidence / loss calculation process involved in the present application.

[0024] Figure 3 A schematic diagram of the K-hop subgraph extraction and graph organization process involved in the present application.

[0025] Figure 4 A schematic diagram of the three types of QA (atomic / aggregated / multi-hop) involved in the present application. DETAILED DESCRIPTION

[0026] In the following text, only some exemplary embodiments are simply described. As those skilled in the art can realize, the described embodiments can be modified in various different ways without departing from the spirit or scope of the embodiments of the present application. Therefore, the drawings and the description are considered to be exemplary in nature rather than limiting.

[0027] The following disclosure provides many different embodiments or examples for implementing different structures of the embodiments of the present application. In order to simplify the disclosure of the embodiments of the present application, the components and settings of specific examples are described in the following text. Of course, they are only examples and the purpose is not to limit the embodiments of the present application. In addition, the embodiments of the present application can refer to the same reference numerals and / or reference letters in different examples, and such repetition is for the purpose of simplification and clarity, which does not indicate the relationship between the various embodiments and / or settings discussed.

[0028] The embodiments of the present application are described in detail below in conjunction with the drawings.

[0029] The present embodiment provides a knowledge graph-based high-quality instruction data generation method in the field of shipbuilding, which is used to provide high-quality instruction data for the supervised fine-tuning training of large language models in specific fields. The method comprises: (1) Collecting multi-source heterogeneous text data in a specific field, preprocessing the multi-source heterogeneous text data to obtain normalized text corpus; (2) Automatically extracting domain key entities and their semantic relationships from the normalized text corpus, merging duplicate entities and relationships, and using confidence or manual verification to solve conflicting information, and constructing an extensible domain knowledge graph; the domain knowledge graph is composed of a set of key entities and a set of relationships; (3) Using a training model to understand and evaluate each knowledge point in the domain knowledge graph, and calculate the understanding loss of each knowledge point; according to the understanding loss threshold, mark weak knowledge points and mastered knowledge points; (4) taking the relationship in the field knowledge graph as a starting edge, based on the understanding loss, the depth constraint and the length constraint, a subgraph is extracted by using a specified traversal strategy to obtain a target subgraph; (5) based on the structural features of the target subgraph, three types of instruction-response pairs of atomic type, aggregation type and multi-hop type are automatically generated; (6) the update changes of the data source are listened to, the pre-processing and knowledge extraction processes of steps (1)-(2) are repeated for the newly added or modified text data, and the incremental knowledge is incorporated into the field knowledge graph to form a difference list containing new relationships, modified relationships and invalid relationships; (7) for the new relationships and modified relationships in the difference list, new instruction-response pairs are generated by repeating steps (3)-(5), the original instruction-response pairs corresponding to the invalid relationships are marked or deleted, and the new instruction-response pairs are quality audited; (8) all instruction-response pairs are subjected to multi-dimensional quality evaluation, and unqualified samples are filtered or regenerated, and finally a qualified high-quality supervision fine-tuning training data set is output.

[0030] In some embodiments, the multi-source heterogeneous text data specifically includes enterprise internal documents, industry standard specifications, public papers, patent documents, professional assessment standards, design specifications and process procedures of a specific field; the preprocessing specifically includes: format analysis and pure text extraction on the multi-source heterogeneous text data, removing header and footer, repeated page numbers, invalid symbol noise, sentence or paragraph segmentation, and implementing term standardization processing on synonyms and abbreviations.

[0031] In some embodiments, the field key entities in step (2) include parts, process flow, materials and standard specifications; the semantic relationships include part-function, part- composition, process-step, process-used material, process-followed standard, and part-applicable specification; the construction of the field knowledge graph further includes introducing a field ontology model, mapping the extracted entities to predefined categories, limiting available relationship patterns, and writing entities and relationships into a graph database to realize unified storage and query.

[0032] In some embodiments, the specific process of understanding evaluation in step (3) is: each relationship in the field knowledge graph is converted into a natural language statement, a plurality of restatement forms and their negative forms of the statement are generated by using a synthesizer model, an is / yes judgment prompt is constructed and input into a training model, a judgment confidence output by the training model is obtained, a relationship confidence and an understanding loss of the relationship are calculated based on the confidence, and when the understanding loss is higher than a preset threshold, it is marked as a weak knowledge point, otherwise it is marked as a mastered knowledge point.

[0033] In some embodiments, the depth constraint in step (4) is to limit the maximum hop number of subgraph extraction, and the length constraint is to limit the upper limit of the length of the premise text corresponding to the subgraph; the specified traversal strategy includes three types: a strategy of preferentially selecting edges with high understanding loss, a strategy of preferentially selecting edges with low understanding loss, and a strategy of uniformly and randomly selecting edges, and a single strategy or a combined strategy is selected according to training requirements.

[0034] In some embodiments, the atomic instruction-response pair in step (5) is generated based on a single-edge or single-node subgraph, and a question and answer are constructed for a single knowledge point; the aggregated instruction-response pair is generated based on a subgraph containing multiple association relationships of the same topic domain, and after the association relationships are restated, compressed and summarized by a synthesizer model to form a natural language answer, a corresponding question is generated in reverse; the multi-hop instruction-response pair is generated based on a subgraph across multiple relationships, and a question requiring multi-step reasoning is constructed, and the answer depends on the combination of multiple knowledge chains.

[0035] In some embodiments, the update of the data source in step (6) specifically includes text updates corresponding to new specifications in a specific field, modified processes, and introduced new materials; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed part is supplemented or corrected, and the same standard as the multi-dimensional quality evaluation in step (8) is used for the quality audit.

[0036] In some embodiments, the multi-dimensional quality evaluation in step (8) includes language dimension, knowledge dimension and structure dimension, wherein the language dimension evaluates grammatical correctness, term standardization and expression clarity, the knowledge dimension evaluates the consistency and non-illusion of the answer with the facts in the knowledge graph, and the structure dimension evaluates the distribution of question types, difficulty levels and repetition rates; the quality evaluation also includes comprehensive evaluation, which combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on a closed-book knowledge-intensive task.

[0037] In some embodiments, when generating the instruction-response pair, language diversity is improved by: designing multiple template questions for each semantic relationship, using a large language model to rewrite the template questions for multiple rounds, forming multiple questions with the same semantic meaning but different expressions, and in aggregated and multi-hop instruction-response pairs, combining multiple knowledge points into a single question to improve comprehensive understanding difficulty.

[0038] In some embodiments, the specific field is the shipbuilding field, and the domain knowledge graph constructed in step (2) includes a welding process-step-material used-inspection standard subdomain, and the instruction-response pairs generated in step (5) involve atomic, aggregated and multi-hop instruction-response pairs related to welding method operation steps, welding quality judgment standards and defect treatment measures.

[0039] The technical concept of the present application is as follows: The present application constructs a technical scheme around "knowledge graph driven SFT instruction data automatic synthesis in the field of shipbuilding", and the core process is "multi-source data acquisition - field knowledge graph construction - understanding evaluation and graph organization - instruction sample generation - automatic update and quality evaluation": First, collect multi-source unstructured / semi-structured texts such as internal documents of shipbuilding enterprises, industry standards, public papers and patents, and form normalized corpus after pretreatment; Based on natural language processing technology and pre-trained large models, extract key entities (parts, process flow, materials, standards, etc.) and semantic relationships (part-function, process-step, process-standard, etc.) from the corpus, and construct an extensible shipbuilding knowledge graph; Through training model, score the "understanding degree" of each knowledge point in the knowledge graph, calculate the understanding loss, combine K-hop subgraph extraction algorithm and multiple traversal strategies, and select the target subgraph; Based on the subgraph structure, automatically generate three types of instruction-response pairs: atomic (single knowledge point question and answer), aggregation (multi-associated knowledge point summary / contrast question and answer), and multi-hop (multi-step reasoning question and answer), and improve language diversity through template rewriting; An automatic updating mechanism is established to listen to data source changes and incrementally update the knowledge graph and data set, while performing double quality evaluation on the synthesized data through language, knowledge, and structure multi-dimensional indicators and downstream model performance to ensure that the data meets the standards.

[0040] The purpose of the present application is to propose a high-quality supervised fine-tuning (SFT) data construction method for the field of shipbuilding, which can fully utilize the knowledge graph to structure the field knowledge, and automatically generate multi-type (atomic, aggregation, multi-hop) instruction question and answer samples through templating and diversification strategies, to solve the problems of high cost, high illusion rate and low coverage of traditional data construction.

[0041] The specific purposes of the present application include: A mechanism for extracting professional knowledge from multi-source unstructured text and constructing an extensible knowledge graph is provided; A mechanism for automatically generating diversified and high-quality instruction-response pairs based on graph structure is provided, which is used for fine-tuning language models; A graph traversal and subgraph organization strategy is provided, which combines entity relationship understanding degree and preferentially covers weak knowledge points; An automatic updating mechanism for instruction data set is provided to ensure that model training data is dynamically synchronized with knowledge evolution; A complete data quality evaluation system is provided to ensure that the generated content meets high standards in terms of accuracy, expressiveness and consistency.

[0042] Through the implementation of the above target, the artificial labeling cost can be significantly reduced, the coverage, reliability and timeliness of model training data are improved, and strong support is provided for the professional landing of large language models in the industrial field.

[0043] As Figure 1 shown, the synthetic algorithm takes the original corpus in the field of shipbuilding as input, and obtains a structured synthetic instruction data set through a three-stage process of "extraction-organization-generation". The formal representation is as follows: ; : self-enterprise internal documents, industry standards, public papers, patents and other multi-source original texts in the field of ships; : represents "knowledge extraction operation", which is the process of extracting information from and building a knowledge graph; KG=(E,R): Ship field knowledge graph composed of key entity set E and relation set R; : is "data organization operation", that is, the process of converting structured knowledge into instruction-response pairs based on KG; : the finally generated SFT training data in the form of high-quality instruction-response (prompt-response).

[0044] The method is composed of the following main functions: 1. Data source and collection: responsible for multi-source text data acquisition and preprocessing; 2. Domain knowledge graph construction: responsible for entity recognition, relation extraction and ontology modeling, forming an extensible KG; 3. Understanding evaluation: scoring the "understanding degree" of each knowledge point through the trained model ; 4. Subgraph extraction and graph organization: K-hop subgraph extraction based on loss constraint and length constraint; 5. Instruction generation: generating atomic / aggregated / multi-hop QA based on entity-relation patterns and subgraph structure; 6. Automatic update: listening to data source changes and performing incremental update on KG and QA set; 7. Quality evaluation: using multi-dimensional indicators to evaluate the internal quality and downstream effect of .

[0045] The comprehensive quality index Q can be formalized as: ; Where Quality of data itself, Quality of model after SFT Performance on closed-book knowledge-intensive tasks; Metric: refers to a set of multi-dimensional quality evaluation indicators (a set of pre-defined indicators), including language dimensions (grammatical correctness, term standardization, expression clarity), knowledge dimensions (fact consistency, no illusion), and structure dimensions (question type distribution, difficulty level distribution, repetition rate) specific quantitative indicators; : refers to the test data set (i.e. evaluation data set) used to evaluate the performance of the model; : refers to the target large language model fine-tuned by Dsynth generated by the present application (i.e. trained model).

[0046] ; Wherein, L is the language dimension score (value range 0-100), which is calculated by weighted calculation of grammatical correctness (weight 0.3), term standardization (weight 0.4), and expression clarity (weight 0.3); K is the knowledge dimension score (value range 0-100), which is calculated by weighted calculation of fact consistency (weight 0.7) and no illusion rate (weight 0.3); S is the structure dimension score (value range 0-100), which is calculated by weighted calculation of question type distribution rationality (weight 0.3), difficulty level distribution uniformity (weight 0.3), and sample repetition rate (reverse scoring, weight 0.4); is the weight of each dimension, satisfying The default values are 0.3, 0.5, and 0.2, respectively, which can be adjusted according to the requirements of the field.

[0047] ; Wherein, P is the precision (Precision), R is the recall (Recall), and F1 is the F1 score, which are calculated based on the question and answer accuracy of the model on ; α, β, γ are the weights of the indicators, satisfying α+β+γ=1, and the default values are all 1 / 3, which can be adjusted according to the task emphasis (such as prioritizing answer accuracy or completeness).

[0048] Each functional structure and working principle: Data sources and collection: Data sources include: Internal documents of shipbuilding enterprises: such as ship structure design instructions, cabin layout diagram instructions, cable laying regulations, welding / coating process cards, etc.; Industry standards and specifications: such as ship classification society specifications, welding process evaluation standards, material selection standards, etc.; Publications and patents: Research results on ship structure optimization, process automation, intelligent inspection, etc.

[0049] The above unstructured / semi-structured text is uniformly preprocessed, including: Format analysis and pure text extraction; Remove headers and footers, repeated page numbers, invalid symbols, and other noise; Sentence / paragraph segmentation and term normalization (synonyms, abbreviations).

[0050] The resulting normalized text corpus is used as input for subsequent knowledge extraction.

[0051] Domain knowledge graph construction: With the preprocessed corpus as input, automatically extract key entities and their relationships in the shipbuilding domain, and construct a domain KG. Key entities include but are not limited to: Parts: hull, hatch cover, main engine, auxiliary machinery, welded parts, support parts, etc. Process flow: blanking, cutting, plate assembly, three-dimensional assembly, welding, flaw detection, etc. Materials: various ship steels, alloy materials, welding materials, coatings, etc. Standard specifications: such as a ship classification society specification clause, process evaluation standard number, etc.

[0052] Relationships represent semantic associations between entities, such as: Part-function; Part composition; Process step; Process materials; Process standards; Part-applicable specifications, etc.

[0053] To achieve this, we combine natural language processing techniques with pre-trained large models: 1. Use LLM Parse paragraph text into triple-form knowledge statements; 2. Introduce a domain ontology model to map entities to predefined categories and limit available relationship patterns (e.g. "part-function" "process-step" etc.); 3. Merge entities and relationships that appear repeatedly across segments, and use confidence or manual verification to resolve conflicting information; 4. Write the resulting entities and relationships to a graph database to enable unified storage and querying.

[0054] In this way, the originally scattered knowledge distributed in long documents in natural language is converged into a structured graphical representation, significantly reducing the illusion probability in the subsequent generation process, and providing a basis for automatic reasoning.

[0055] Understanding evaluation and relationship confidence calculation (corresponding Figure 2 ): In order to make the generated data more targeted, the present application proposes to train the model The "understanding degree" on each knowledge point is explicitly measured. For each relationship , its natural language description can be regarded as a statement , and the true probability is theoretically 1: ; A set of restated statements and their negative forms are generated for , and the confidence of the "yes / no" judgment given by is prompted, thereby defining: ; : the confidence of the relationship ; n: the total number of samples involving the relationship ; : the case of "relationship " in the jth sample; : the case of "no relationship " in the jth sample; : the probability of "correct match (t represents true)" when the relationship exists in sample j; : the probability of "wrong match (f represents false)" when the relationship does not exist in sample j; And define the understanding loss: ; : the understanding loss of the relationship (corresponding to the understanding error of the model for the relationship); The understanding loss describes the gap from "completely ignorant" to "completely mastered" of for this knowledge point, providing a quantitative basis for subsequent subgraph extraction and training data selection: the higher the loss, the weaker the model's current area for this knowledge point, and relevant training samples need to be generated first.

[0056] Subgraph extraction and graph organization (corresponding Figure 3 ): ​In the best embodiment, subgraphs are the basis for generating minimal QA units. The invention employs a K-hop subgraph extraction algorithm to perform a constrained traversal of the graph starting from an initial relation : refers to the "source entity", which is the starting entity of the relation instance (such as "hull section" in the shipbuilding field).

[0057] : refers to the "target entity", which is the target entity of the relation instance (such as "welding process" corresponding to "hull section"). The pseudo code is as follows: Input: graph G, initial edge R_i, traversal strategy S; Output: subgraph G'; 1. G' <- {R_i}; 2. Candidate set C <- adjacent edge set ( ); 3. while C ≠ ∅ do; a. Select edge e according to S; b. G' <- G' ∪ {e}; c. Terminate if constraints are met; d. Add adjacent edges of the nodes at both ends of e to C; Return G'; Constraints include two categories: Depth constraint: limit the maximum number of hops K to control the length of the reasoning chain; Length constraint: set an upper limit on the premise token length pre_length of the subgraph to avoid sample distribution imbalance caused by excessively large subgraphs.

[0058] Traversal strategy S supports the following options: 1. max_loss: prefer to select edges with high understanding loss, focusing on generating knowledge that the model "cannot" understand; 2. min_loss: prefer to use edges with low understanding loss to construct stable links as high-quality context; 3. Random: uniformly random selection to ensure coverage.

[0059] Through the above strategy combination, the complexity, relevance, and computational cost of the subgraph are balanced and controlled.

[0060] Instruction generation mechanism (corresponding to Figure 4 ): Based on the constructed knowledge graph and extracted subgraph, the invention designs a mechanism for automatically generating prompt-response pairs.

[0061] 1. Atomic QA When the size of the subgraph is a single edge or a single node, generate questions and answers for a single knowledge point. For example, a triple​ <component X, function, Y> => Q: "What is the function of component X?" A: "The function of component X is Y." 2. Aggregated QA When a subgraph contains multiple relations that are interrelated but belong to the same thematic domain (e.g., multiple steps of a process, multiple components of a device), the invention utilizes a synthesizer model These relations are "restated - compressed - summarized" to generate a complete explanatory natural language answer, and then the corresponding question is automatically generated in reverse.

[0062] Such QA is used to examine the summarization ability, comparison ability, and structured information integration ability of the model, and is suitable for process flow, specification clause, component composition, etc. For example, triplets: <welding process GMAW, step, root cleaning>; <welding process GMAW, step, hot welding>; <welding process GMAW, step, filling>; <root cleaning, tool used, angular abrasive wheel>; The automatically generated restated answer ( output) is as follows: The GMAW gas shielded welding process generally includes three main steps: root cleaning, hot welding, and filling.

[0063] The root cleaning step requires the use of an angular abrasive wheel to polish the root of the weld.

[0064] The automatically generated question is as follows: What steps are included in the GMAW gas shielded welding process? What tools are required for the root cleaning step? The aggregated QA formed is as follows: Q: What steps are included in the GMAW welding process? What tools are required for the root cleaning step? A: The process includes three steps: root cleaning, hot welding, and filling, and the root cleaning step requires the use of an angular abrasive wheel.

[0065] 3. Multi-hop QA For subgraphs that span multiple relations, explicitly construct questions that require multi-step reasoning, making the answer dependent on the combination of multiple knowledge chains, thereby training the model's reasoning ability in complex processes, system composition, causal relationships, etc. For example, triplets, as follows: <cabin bulkhead model X12, material, AH36 steel>; <AH36 steel, applicable welding process, CO2 gas shielded welding>; <CO2 gas shielded welding, inspection standard, ship classification society specification 5.4.2>; This is a typical "material-process-standard" three-hop chain.

[0066] The automatically generated restated answer is as follows: Output:The bulkhead plate X12 adopts AH36 high-strength marine steel, so the applicable welding process is CO2 gas shielded welding.

[0067] According to industry standards, CO2 gas shielded welding should comply with the clause 5.4.2 of the ship classification society standard.

[0068] The automatically generated question is as follows: What welding process should be used for the bulkhead plate X12, and what inspection standards does it need to comply with? The multi-hop QA formed is as follows: Question: What welding process should be used for the bulkhead plate X12, and what inspection standards does it need to comply with? Answer: The applicable welding process for the bulkhead plate X12 is CO2 gas shielded welding, which should comply with the clause 5.4.2 of the ship classification society standard.

[0069] To improve language diversity, the invention: Designs multiple template questions for each type of relationship; Uses LLM to perform multiple rounds of rewriting on the templates to form multiple questions that are semantically equivalent but expressed differently; In the combined question and answer, multiple knowledge points are combined into the same question to improve the difficulty of comprehensive understanding.

[0070] During generation, the system records the source triple ID or subgraph ID for each question and answer sample to support subsequent updates and traceability.

[0071] Automatic update and maintenance: When the data source changes (new specifications, modified processes, introduction of new materials, etc.), the automatic update of the invention is responsible for: 1. Repeating the "preprocessing + knowledge extraction" process for new data to incorporate incremental knowledge into the KG; 2. Comparing the knowledge sets before and after the update to form a difference list (new relationships, modified relationships, and invalid relationships); 3. Generating new QA samples for new / modified relationships through the instruction generation function, and marking or deleting samples related to invalid relationships; 4. Triggering the quality evaluation process to audit the incremental samples.

[0072] Thanks to the binding design of question and answer samples and triples, the entire update process can be completed within the incremental range, without the need to rebuild the entire data set, ensuring real-time synchronization of training data and domain knowledge.

[0073] Synthetic data quality evaluation: For Multi-dimensional scoring, including but not limited to: Language dimension: grammatical correctness, terminology standardization, expression clarity; Knowledge dimension: whether the answer is strictly consistent with the facts in KG, whether there is illusion; Structural dimension: question type distribution, difficulty level distribution, repetition rate, etc.

[0074] And the comprehensive evaluation is as follows: ; Where Metric is a set of pre-defined indicators. Samples that do not meet the indicators will be filtered or regenerated to ensure that high-quality data enters the SFT training.

[0075] Best embodiment: SFT data construction for ship welding process specification In a specific and preferred embodiment, the present application is applied to the training of "hull welding process specification auxiliary question and answer model": 1. Collect texts from the welding process regulations, weld quality inspection standards, welding material selection manual and related specification clauses of a shipyard; 2. Build a knowledge graph sub-domain containing "welding process -> step -> material used -> inspection standard"; 3. Based on this sub-domain, generate a large number of atomic, aggregated and multi-hop questions and answers about "operation steps of a certain welding method", "weld quality judgment standard", "defect treatment measures", etc. 4. After quality screening of the generated data, it is used for SFT training of general Chinese large models; 5. After training, the model can accurately answer welding process related questions under closed-book conditions, and can explain complex scenarios in multiple steps, such as "when the longitudinal butt weld of the hull appears an incomplete penetration defect, how should it be handled according to the specification" and the like.

[0076] Experiments show that after fine-tuning using the data generated by the present application, the professional question and answer accuracy and stability of the model in the welding scene are significantly better than the baseline method using "a small amount of artificial examples + general synthesis".

[0077] Comparison with background technology and improvement points: Compared with the existing methods in the background technology, the present application has significant improvements in the following aspects: 1. From "text-driven" to "graph-driven" data synthesis paradigm Traditional methods directly generate questions and answers based on raw documents or scattered paragraphs, lack explicit modeling of knowledge structure, and are prone to uneven knowledge coverage, long-tail information omission, and fact hallucinations. The present invention constructs a domain knowledge graph, explicitly organizes knowledge points in a graph structure, and uses it as the "skeleton" for data synthesis, greatly improving the systematicness and accuracy of knowledge coverage.

[0078] 2. Data selection mechanism guided by "understanding loss" Existing methods usually do not distinguish between knowledge points that the model has mastered and those that it has not, or only select training samples through rough difficulty estimation. The present invention defines and other metrics to explicitly measure the understanding of each knowledge point, and uses K-hop graph organization strategy to generate more training samples for the model's "short board knowledge", improving training efficiency and effectiveness.

[0079] 3. Unified framework supporting atomic, aggregated, and multi-hop multi-level QA Traditional synthesis methods often only cover basic questions and answers, lacking systematic support for multi-relation reasoning and complex scenarios. The present invention generates atomic QA, aggregated QA, and multi-hop QA based on subgraph size and structure, ensuring comprehensive coverage of basic knowledge and systematically enhancing the model's multi-step reasoning ability, making the fine-tuned model more relevant to engineering practical problems.

[0080] 4. Automatic update and traceable data maintenance mechanism In existing technologies, knowledge update often requires manual reconstruction of data sets, which is costly and prone to errors. The present invention uses a "sample-triplet-subgraph" three-layer binding combined with a differential detection mechanism to achieve incremental update and tracking of synthetic data. With the introduction of new specifications and new technologies, the model can continuously absorb new knowledge through multiple fine-tuning, avoiding the emergence of "outdated answers".

[0081] 5. Dual evaluation system of intrinsic quality and downstream performance Traditional methods rely on manual spot-checking or a small number of automatic indicators to evaluate synthetic data, making it difficult to systematically measure its real value to downstream tasks.

[0082] The present invention measures ; the quality of the data itself and the performance of the fine-tuned model on closed-book knowledge-intensive tasks, ensuring that "good data" can truly lead to "good models".

[0083] Comprehensively, the present application makes systematic improvements on the overall architecture, method flow and evaluation system of the prior art, realizes the automatic construction of SFT instruction data facing the ship manufacturing field, high quality and sustainable evolution, and has significant engineering practical value and popularization prospect.

[0084] The above-described embodiments are intended to illustrate the present application, and are not intended to limit the present application, so that the change of example values or the replacement of equivalent elements should still belong to the scope of the present application.

[0085] From the above detailed description, it can be seen that the present application can achieve the above-mentioned purposes, and has met the requirements of the Patent Law.

[0086] Although the preferred embodiments of the present application have been described, those skilled in the art can make further changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all changes and modifications falling within the scope of the present application. The above description is only the preferred embodiments of the present application and is not intended to limit the present application. It should be noted that any modification, equivalent replacement and improvement made within the spirit and principle of the present application should be included in the protection scope of the present application.

[0087] It should be noted that the above description of the flow is only for example and illustration, and does not limit the scope of the present application. Those skilled in the art can make various modifications and changes to the flow under the guidance of the present application. However, these modifications and changes are still within the scope of the present application.

[0088] The above has described the basic concept, and it is obvious that the above-mentioned invention disclosure is only as an example and does not constitute a limitation on the present application for those skilled in the art after reading this application. Although it is not explicitly stated here, those skilled in the art can make various modifications, improvements and modifications to the present application. Such modifications, improvements and modifications are suggested in the present application, so such modifications, improvements and modifications still belong to the spirit and scope of the exemplary embodiments of the present application.

[0089] Meanwhile, specific words are used in the present application to describe the embodiments of the present application. For example, "one embodiment", "an embodiment", and / or "some embodiments" means a certain feature, structure or characteristic related to at least one embodiment of the present application. Therefore, it should be emphasized and noted that the "an embodiment" or "one embodiment" or "an alternative embodiment" mentioned in different positions in the specification does not necessarily refer to the same embodiment. In addition, some features, structures or characteristics in one or more embodiments of the present application can be properly combined.

[0090] Moreover, as will be appreciated by persons skilled in the art, the present application is capable of being embodied with several alternatives now known or later developed in the art, including any newly discovered equivalents, which do not depart from the scope and spirit of the present application. Accordingly, it is not intended that the application be limited as described herein, but rather that the application be limited only as set forth in the claims. For example, although the implementation of various components described above can be embodied in hardware, it can also be implemented as a software only solution (e.g., a computer program product which can include a computer readable medium having stored computer program code thereon). Such software can be loaded into memory of a computer system and executed by a processor of the computer system to implement the functions of the various components described herein. The computer program code can also be stored in a computer readable medium that can be a transitory or non-transitory medium known in the art, including for example: any combination of data storage media conditions above. As such, various computer program products can be created using the computer program code.

[0091] Computer program code for carrying out operations of the present application can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Scala, Smalltalk, Eiffel, JADE, Emerald, C++, C#, VB.NET, Python and the like, conventional procedural programming languages, such as the C programming language, Visual Basic, Fortran 2103, Perl, COBOL 2102, PHP, ABAP, dynamic programming languages, such as Python, Ruby and Groovy, or another programming language. The program code can execute entirely on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider). In some embodiments, electronic program guide data, program schedule data, and / or other data can be obtained via data communication from a service provider through a wired connection, such as a modem or other wired coupling, or interface to a network adapter. The data can also be obtained with a wireless connection, such as over the airwaves of a cellular telephone network, or other mobile network. In some embodiments, the data can be obtained via a cloud computing environment, or as a service, such as Software as a Service (SaaS).

[0092] Moreover, the order of processing elements or sequence of steps in which elements of the application are carried out, as previously discussed, is not limiting of the present application, unless expressly stated otherwise. Although the above disclosure discusses several exemplary embodiments of the application, it should be appreciated that structural and functional modifications can be made without departing from the scope and spirit of the application. For example, although the implementation of various components described above can be embodied in hardware, it can also be implemented as a software only solution (e.g., a computer program product which can include a computer readable medium having stored computer program code thereon). Such software can be loaded into memory of a computer system and executed by a processor of the computer system to implement the functions of the various components described herein. The computer program code can also be stored in a computer readable medium that can be a transitory or non-transitory medium known in the art, including for example: any combination of data storage media conditions above. As such, various computer program products can be created using the computer program code.

[0093] For similar reasons, it is to be appreciated that the teachings of the present application provided herein can be applied to any embodiment of the present application, and that actual claims applied for or patent granted can be broader than any single, featured embodiment. Accordingly, a patent applicant has constructed and filed examples to particularly point out and distinctly claim those aspects which are regarded as novel and those aspects specifically shown.

Claims

1. A knowledge graph-based method for generating high-quality instruction data in the field of shipbuilding, characterized by, The method comprises the following steps: (1) collecting multi-source heterogeneous text data in a specific field, preprocessing the multi-source heterogeneous text data to obtain normalized text corpus; (2) automatically extracting domain key entities and semantic relationships from the normalized text corpus, merging repeated entities and relationships, solving conflict information by confidence or manual verification, and constructing an extensible domain knowledge graph; (3) using a trained model to evaluate and assess each knowledge point in the domain knowledge graph, and calculating the understanding loss of each knowledge point; (4) taking the relationship in the domain knowledge graph as the starting edge, based on the understanding loss, depth constraint and length constraint, using a specified traversal strategy to extract subgraphs to obtain target subgraphs; (5) based on the structural features of the target subgraph, automatically generating three types of instruction-response pairs, including atomic type, aggregation type and multi-hop type; (6) listening to the update of the data source, repeating the preprocessing and knowledge extraction process of steps (1)-(2) for the newly added or modified text data, and incorporating the incremental knowledge into the domain knowledge graph to form a difference list containing new relationships, modified relationships and invalid relationships; (7) generating new instruction-response pairs by repeating steps (3)-(5) for the new relationships and modified relationships in the difference list; (8) multi-dimensional quality evaluation is performed on all instruction-response pairs, and unqualified samples are filtered or regenerated, and finally a qualified high-quality supervised fine-tuning training data set is output. 2.The knowledge graph-based high-quality instruction data generation method for the shipbuilding field according to claim 1, characterized in that, The multi-source heterogeneous text data specifically includes enterprise internal documents, industry standards, public papers, patent documents, professional evaluation standards, design specifications and process procedures in a specific field; the preprocessing specifically includes format analysis and pure text extraction of multi-source heterogeneous text data, removing header and footer, repeated page numbers, invalid symbol noise, sentence or paragraph segmentation, and implementing term standardization processing on synonyms and abbreviations. 3.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, The domain key entities in step (2) include parts, process flow, materials and standard specifications; the semantic relationships include part-function, part-constitution, process-step, process-used material, process-followed standard, and part-applicable specification; the construction of the domain knowledge graph also includes introducing a domain ontology model, mapping the extracted entities to predefined categories, limiting available relationship patterns, and writing entities and relationships into a graph database to realize unified storage and query. 4.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, The specific process of understanding evaluation in step (3) is as follows: each relationship in the domain knowledge graph is converted into a natural language statement, a synthesizer model is used to generate multiple restatements and their negative forms of the statement, an is / yes judgment prompt is constructed and input into a trained model, a judgment confidence is obtained from the trained model output, the relationship confidence and understanding loss of the relationship are calculated based on the confidence, and when the understanding loss is higher than a preset threshold, it is marked as a weak knowledge point, otherwise it is marked as a mastered knowledge point. 5.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, The depth constraint in step (4) is to limit the maximum hop number of subgraph extraction, and the length constraint is to limit the upper limit of the length of the premise text corresponding to the subgraph; the specified traversal strategy includes three types: a strategy of preferentially selecting edges with high understanding loss, a strategy of preferentially selecting edges with low understanding loss, and a strategy of uniformly and randomly selecting edges, and a single strategy or a combined strategy is selected according to training requirements. 6.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, In step (5), the atomic instruction-response pair is generated based on a single-edge or single-node subgraph, and a question and answer are constructed for a single knowledge point; the aggregated instruction-response pair is generated based on a subgraph containing multiple association relationships of the same topic domain, and after the synthesizer model restates, compresses and summarizes the association relationships to form a natural language answer, the corresponding question is generated in reverse; the multi-hop instruction-response pair is generated based on a subgraph across multiple relationships, and a question requiring multi-step reasoning is constructed, and the answer depends on the combination of multiple knowledge chains. 7.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, In step (6), the update of the data source specifically includes text updates corresponding to new specifications in a specific field, modified processes, and introduced new materials; when the incremental knowledge is incorporated into the domain knowledge graph, the integrity of the original valid entities and relationships is maintained, and only the changed part is supplemented or corrected, and the quality audit and the multi-dimensional quality evaluation in step (8) use the same standard. 8.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, In step (8), the multi-dimensional quality evaluation includes language dimension, knowledge dimension and structure dimension, wherein the language dimension evaluates grammatical correctness, terminology standardization and expression clarity, the knowledge dimension evaluates the consistency and non-illusion of the answer with the facts in the knowledge graph, and the structure dimension evaluates the problem type distribution, difficulty level distribution and repetition rate; the quality evaluation also includes comprehensive evaluation, and the comprehensive evaluation combines the quality of the instruction-response pair itself and the performance of the model after fine-tuning using the instruction-response pair on a closed knowledge-intensive task. 9.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 6, characterized in that, When generating the instruction-response pair, the language diversity is improved by: designing multiple template questions for each semantic relationship, using a large language model to rewrite the template questions for multiple rounds, forming multiple questions with the same semantics but different expressions, and in the aggregated and multi-hop instruction-response pairs, combining multiple knowledge points into a single question to improve comprehensive understanding difficulty. 10.The knowledge graph based high-quality instruction data generation method for shipbuilding field according to claim 1, characterized in that, The specific field is the shipbuilding field, and the domain knowledge graph constructed in step (2) includes a welding process-step-material used-inspection standard subdomain, and the instruction-response pairs generated in step (5) involve atomic, aggregated and multi-hop instruction-response pairs related to welding method operation steps, weld quality judgment standards and defect treatment measures.

Images