Dataset generation methods, devices, equipment, and media for models in the field of insulating gases
By parsing data from the insulating gas domain into corpus units and constructing candidate corpus fragments, and combining domain label trees and generation task constraints, the problems of high manual costs and sample inconsistency in the insulating gas model are solved, realizing the automated construction of high-quality datasets and improving the reliability and practicality of the model.
Patent Information
- Application Number
- CN202611207292.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-10
- Publication Date
- 2026-11-17
AI Technical Summary
Existing technologies face challenges in building models for insulating gases, including high labor costs, inconsistent sample styles, uneven coverage of question types, and difficulty in ensuring semantic consistency, which limits the reliability and practicality of the models.
Technical data in the field of insulating gases is parsed into corpus units and processed into candidate corpus fragments through semantic segmentation. A label tree for the field of insulating gases is introduced, and question-answer pairs are constructed based on the label range and constraints of the generation task to achieve automated construction of the dataset.
It improves the domain specialization and accuracy of the dataset, enhances the reliability and practicality of the insulating gas domain model, increases the flexibility and reusability of dataset construction, and reduces labor costs.
Smart Images

Figure CN123115368A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of data processing, and in particular to a method, apparatus, device and medium for generating datasets for insulating gas models. Background Technology
[0002] Insulating gases are widely used in high-voltage electrical equipment such as gas-insulated switchgear, transmission lines, and circuit breakers. The industry has accumulated a wealth of professional data on the insulation and arc-quenching properties, decomposition products, safety, and environmental impact of sulfur hexafluoride (SF6) and its alternative gases, including standards, test reports, papers, patents, equipment manuals, and maintenance records. This data is highly specialized, scenario-dependent, and parameter-intensive, making it a key data source for constructing large-scale linguistic models in the field of insulating gases. While general-purpose models are powerful, high-quality professional data is still needed to support fine-tuning, retrieval enhancement, or effect evaluation in vertical scenarios such as the analysis of the physicochemical properties of insulating gases, equipment operation and maintenance, and arc-quenching mechanisms.
[0003] However, the aforementioned data is scattered across test reports, operation manuals, international standards, safety data sheets, inspection images, and expert experience, and generally suffers from problems such as inconsistent formats, uneven structures, dense tables, and excessive noise, making it difficult to directly convert into a training or evaluation dataset.
[0004] Existing processing methods mostly rely on manually extracting key information such as gas composition, test conditions, and breakdown voltage and compiling samples. This approach is only suitable for small-scale data. When dealing with large-scale data that spans standards, devices, and gas systems, it suffers from limitations such as high manual costs, inconsistent sample styles, and uneven coverage of question types. Furthermore, it is difficult to ensure semantic consistency between questions, answers, and the original sources, which restricts the reliability and practicality of models in the field of insulating gases. Summary of the Invention
[0005] In view of this, this disclosure proposes a dataset generation method, apparatus, device, and medium for models in the field of insulating gases. This method organizes technical data related to the insulating gas field into unified candidate corpus fragments. Based on the label range and generation constraints of the current generation task, it constructs datasets adapted to different generation tasks using the candidate corpus fragments as the data foundation. This achieves automated construction of the target dataset from professional data related to the insulating gas field, avoiding the manual cost of extracting information and writing samples in traditional techniques. The constructed dataset's question-answer pairs follow entity knowledge or experimental facts in the insulating gas field, maintaining semantic consistency between sample questions, sample answers, and the original information in the source documents. Simultaneously, by utilizing the label range and generation constraints of the current generation task, it avoids the limitations of inconsistent sample styles and unbalanced question type coverage caused by manual writing, thereby improving the domain specialization and accuracy of the dataset and enhancing the reliability and practicality of the insulating gas field model.
[0006] According to one aspect of this disclosure, a method for generating a dataset for an insulating gas domain model is provided, the method comprising:
[0007] For each original technical document related to the field of insulating gases, the original technical document is parsed into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique identifier of the original technical document and the position information of the corpus unit in the original technical document;
[0008] The corpus content of each corpus unit parsed from the original technical data is semantically segmented into at least one candidate corpus fragment; any candidate corpus fragment represents a structural-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gas; the candidate corpus fragment carries fragment source information; the fragment source information is determined based on the corpus unit to which the corpus content in the candidate corpus fragment belongs and the unit source information it carries;
[0009] Determine the corpus tags for each candidate corpus segment; the corpus tags include at least one tag node in the insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize the knowledge keywords or technical topics of the insulating gas domain.
[0010] Based on the label range of the current generation task, candidate corpus fragments whose labels match the label range are selected, and multiple question-answer pairs are constructed based on the generation constraints of the current generation task and the selected candidate corpus fragments; the current generation task is determined based on the application requirements of the model.
[0011] Based on the multiple question-answer pairs already constructed under the current generation task, construct the target dataset corresponding to the current generation task.
[0012] According to another aspect of this disclosure, a dataset generation apparatus for an insulating gas domain model is provided, the apparatus comprising:
[0013] The data parsing module is configured to parse each original technical document related to the field of insulating gases into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique identifier of the original technical document and the position information of the corpus unit in the original technical document.
[0014] The corpus segmentation module is configured to semantically segment the corpus content of each corpus unit parsed from the original technical data into at least one candidate corpus fragment; any candidate corpus fragment represents a structural-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gases; the candidate corpus fragment carries fragment source information; the fragment source information is determined based on the corpus unit to which the corpus content in the candidate corpus fragment belongs and the unit source information it carries;
[0015] The corpus tag determination module is configured to determine the corpus tag for each candidate corpus segment; the corpus tag includes at least one tag node in an insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize a knowledge keyword or technical topic in the insulating gas domain;
[0016] The question-answer pair construction module is configured to filter out candidate corpus fragments whose labels match the label range of the current generation task, and construct multiple question-answer pairs based on the generation constraints of the current generation task and the filtered candidate corpus fragments; the current generation task is determined based on the application requirements of the model.
[0017] The dataset export module is configured to construct the target dataset corresponding to the current generation task based on multiple question-answer pairs already constructed under the current generation task.
[0018] According to another aspect of this disclosure, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.
[0019] According to another aspect of this disclosure, a non-volatile computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the steps of the above-described method.
[0020] In this embodiment, the original technical data is parsed into corpus units with independent semantics and carrying unit source information. The corpus units are then semantically segmented into candidate corpus fragments carrying specific knowledge statements or experimental facts. This effectively solves the semantic fragmentation problem caused by the mixing of charts, formulas, parameters, and text in the literature of insulating gases. A unified candidate corpus fragment is constructed as a basic data resource, overcoming the problem that the heterogeneous technical data in the field of insulating gases is difficult to directly convert into a trainable or evaluable dataset, ensuring the integrity and traceability of the smallest knowledge granularity. Furthermore, an insulating gas domain label tree is introduced based on the organization and application requirements of domain knowledge. Corpus labels are configured for candidate corpus fragments. Candidate corpus fragments are dynamically selected based on the label range and generation constraints of the current generation task. The selected candidate corpus fragments are used as factual basis to construct question-answer pairs according to the generation constraints of the current generation task. This ensures that the constructed question-answer pairs follow the entity knowledge or experimental facts of the insulating gas domain, maintaining the semantic consistency between sample questions, sample answers, and the original information of the source document. This method automates the construction of target datasets from professional data related to insulating gases, avoiding the manual costs of extracting information and writing samples in traditional techniques. At the same time, by using the label range and generation constraints of the current generation task, it avoids the limitations of inconsistent sample styles and uneven question type coverage caused by manual writing, thereby improving the domain professionalism and accuracy of the dataset and enhancing the reliability and practicality of the insulating gas domain model.
[0021] In addition, since different technical data related to insulating gases are processed into a unified candidate corpus fragment, the parsing and labeling process of this candidate corpus fragment is independent of the specific generation task. The same set of candidate corpus fragments can be quickly adapted to the dataset construction requirements of different downstream tasks such as insulating gas composition analysis, fault diagnosis, and material selection. This avoids the computational cost of repeatedly parsing data for each new task, improves the flexibility and reusability of dataset construction, and greatly enhances the efficiency of iterative development of domain models.
[0022] Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description
[0023] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this disclosure together with the specification and serve to explain the principles of this disclosure.
[0024] Figure 1 This application exemplarily illustrates a flowchart of a dataset generation method for a model of insulating gases;
[0025] Figure 2This application exemplarily illustrates a flowchart of semantic segmentation processing of each corpus unit parsed from the original technical data to obtain candidate corpus fragments;
[0026] Figure 3 This application exemplarily illustrates a flowchart for constructing multiple question-answer pairs based on selected candidate corpus fragments;
[0027] Figure 4 This application exemplarily illustrates a flowchart of a question-answer pair targeted supplementation step based on sample balancing;
[0028] Figure 5 This application exemplarily illustrates a flowchart of a process for constructing a target dataset;
[0029] Figure 6 This application exemplarily illustrates a virtual device block diagram of a dataset generation apparatus for an insulating gas domain model;
[0030] Figure 7 This application exemplarily illustrates a block diagram of an electronic device for generating a dataset for an insulating gas domain model. Detailed Implementation
[0031] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.
[0032] As used herein, the terms “comprising,” “including,” “having,” or variations thereof are open-ended and include one or more of the stated features, integrals, elements, steps, components, or functions, but do not exclude the presence or addition of one or more other features, integrals, elements, steps, components, functions, or groups thereof.
[0033] When an element is referred to as “connected,” “coupled,” “responding,” or a variation thereof relative to another element, it may be directly connected, coupled, or responding to another element, or there may be an intermediate element present.
[0034] Although the terms first, second, third, etc., may be used herein to describe various elements / operations, these elements / operations should not be limited by these terms. These terms are only used to distinguish one element / operation from another. Therefore, without departing from the teachings of the inventive concept, a first element / operation in some embodiments may be referred to as a second element / operation in other embodiments.
[0035] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.
[0036] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.
[0037] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, data stored, data displayed, etc.) and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant regions.
[0038] Insulating gases, also known as gaseous insulating media, refer to gaseous materials used in electrical equipment to isolate conductive components and prevent current leakage and arcing. Examples include sulfur hexafluoride. Nitrogen and perfluoroisobutyronitrile New environmentally friendly alternative gases typically possess high electrical strength, good thermal stability, and chemical inertness.
[0039] Insulating gases are widely used in gas-insulated switchgear, gas-insulated transmission lines, circuit breakers, instrument transformers, and related high-voltage electrical equipment. This is particularly relevant to sulfur hexafluoride (SF6). The insulation properties, arc-quenching properties, decomposition products, material compatibility, toxicity, safety, and environmental impact of insulating gases and their alternative gases have been extensively studied, resulting in a wealth of industry standards, test reports, papers, patents, equipment manuals, maintenance records, and case studies. This information is highly specialized, context-dependent, and parameter-dependent, making it a crucial data source for constructing comprehensive linguistic models in the field of insulating gases.
[0040] While insulating gas domain models, such as large language models, have demonstrated strong capabilities in general knowledge tasks, they still require high-quality professional domain data for fine-tuning, retrieval enhancement, generation, or effect evaluation in vertical scenarios such as the analysis of the physicochemical properties of insulating gases in power systems, the operation and maintenance of gas-insulated metal-enclosed switchgear, and high-voltage arc extinguishing mechanisms.
[0041] However, data in the field of insulating gases are usually scattered in physical and chemical test reports, equipment operation manuals, international standards and specifications (such as IEC standards), safety data sheets (MSDS), inspection images and expert experience data. They are inconsistent in format, structure, dense with professional parameters and tables, and have a lot of noise, making it difficult to directly convert them into training or evaluation datasets.
[0042] For insulating gas scenarios, high-quality samples should not only include text-based question-and-answer formats but also cover dimensions such as gas category, mixing ratio, test conditions, equipment, fault phenomena, test results, environmental attributes, assessment question types, and image-based question-and-answer formats. Different downstream applications also have varying requirements for data formats. For example, in model fine-tuning scenarios, insulating gas datasets need to include single-turn question-and-answer formats, command responses, multi-turn maintenance dialogues, expert explanations, and answers tailored to different audiences. In retrieval enhancement generation scenarios, samples need to retain standard clauses, test numbers, page numbers, paragraphs, image identifiers, and contextual windows. In model evaluation scenarios, true / false questions, multiple-choice questions, short-answer questions, open-ended questions, reference answers, and scoring rules are required. The significant differences in data formats for different purposes mean that without a unified generation chain, multiple mutually exclusive data pools with fragmented labeling systems can easily form, making data pool reuse and iteration difficult.
[0043] Furthermore, the field of insulating gases has corresponding requirements for data quality. For example, answers must be supported by source data, and conclusions from different gas systems cannot be mixed; test conditions cannot be divorced from breakdown results; decomposition products, detection methods, and safety conclusions cannot be divorced from equipment status; and labels cannot be concentrated in one place for an extended period. If low-quality samples, such as those using a single alternative gas, are included in the training or evaluation process, the professional reliability of the model will be affected.
[0044] Given the dispersed and modal-diverse nature of technical data in the field of insulating gases, existing technologies for constructing datasets in this area often employ manual compilation. This involves extracting key information such as gas composition, test conditions, breakdown voltage, decomposition products, equipment type, and detection methods from technical data, and then creating question-and-answer pairs, evaluation questions, or training samples based on this information. However, this approach is only suitable for scenarios with small datasets. When dealing with large-scale insulating gas data spanning standards, equipment, and gas systems, it suffers from limitations such as high manual costs, inconsistent sample styles, and uneven question type coverage. Furthermore, it is difficult to ensure semantic consistency between questions, answers, and the original sources, thus restricting the reliability and practicality of models in the field of insulating gases.
[0045] In view of this, this embodiment provides a dataset generation method for insulating gas domain models. This method parses the original technical data into corpus units with independent semantics and carrying unit source information, and then processes the corpus units into candidate corpus fragments carrying specific knowledge statements or experimental facts through semantic segmentation. Furthermore, it introduces an insulating gas domain label tree constructed based on domain knowledge organization and application requirements, configures corpus labels for candidate corpus fragments, and dynamically filters candidate corpus fragments based on the label range and generation constraints of the current generation task. The filtered candidate corpus fragments are used as factual basis to construct question-answer pairs, realizing the automated construction of insulating gas domain-related professional data into the target dataset. This avoids the manual cost of manually extracting information and compiling samples in traditional technologies, improves the domain professionalism and accuracy of the dataset, and enhances the reliability and practicality of insulating gas domain models.
[0046] This method can be executed by electronic devices, such as servers, cloud computing platforms, workstations, personal computers, or edge computing devices deployed in insulating gas laboratories / maintenance sites. It can also be implemented through browser applications, desktop clients, or server-side software. The front-end interface receives project configuration, domain tag correction, and manual review instructions, while the back-end service performs parsing, segmentation, generation, quality control, and export operations. Project information, corpus fragments, tag systems, question-answer pairs, and other information are persistently stored using a database or file system.
[0047] See Figure 1 The flowchart shown is an example of a dataset generation method for an insulating gas domain model. The method may include at least the following steps:
[0048] S110, for each original technical document related to the field of insulating gases, the original technical document is parsed into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique document identifier of the original technical document and the position information of the corpus unit in the original technical document;
[0049] S120, the corpus content of each corpus unit parsed from the original technical data is semantically segmented into at least one candidate corpus fragment; any candidate corpus fragment represents a structural-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gas; the candidate corpus fragment carries fragment source information; the fragment source information is determined based on the corpus unit to which the corpus content in the candidate corpus fragment belongs and the unit source information it carries;
[0050] S130, determine the corpus tag for each candidate corpus segment; the corpus tag includes at least one tag node in the insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize the knowledge keywords or technical topics of the insulating gas domain.
[0051] S140, based on the label range of the current generation task, select candidate corpus segments whose labels match the label range, and construct multiple question-answer pairs based on the generation constraints of the current generation task and the selected candidate corpus segments; the current generation task is determined based on the application requirements of the model.
[0052] S150, Based on the multiple question-answer pairs already constructed under the current generation task, construct the target dataset corresponding to the current generation task.
[0053] Through the above steps, heterogeneous raw data in the field of insulating gases, in terms of source, format, and organization, are unified into standardized candidate corpus fragments. An insulating gas domain tag tree is introduced to label each candidate corpus fragment with its semantic position within the domain knowledge system. This allows massive corpus fragments to be organized and retrieved efficiently according to domain knowledge logic. Furthermore, during the sample construction stage, the tag range and generation constraints are flexibly set based on different model application needs, accurately filtering and calling candidate corpus fragments as needed. This ensures that the constructed question-answer pairs are semantically aligned with the knowledge of the insulating gas domain and meet the specific application requirements of the model in multiple dimensions such as question type distribution, difficulty gradient, expression style, and output format. This achieves configurable and reusable automated construction from raw technical data to high-quality domain question-answer datasets, providing high-quality data for model training and application deployment in the field of insulating gases.
[0054] To enable those skilled in the art to better understand this application, the concepts involved in each step and the specific implementation process of each step will be explained below.
[0055] Regarding step S110, the original technical data related to insulating gases refers to professional data generated throughout the entire lifecycle of insulating gases, including research, production, testing, operation and maintenance, and safety management. This data records information such as the physicochemical properties of insulating gases, equipment application specifications, fault diagnosis logic, and on-site handling experience. It provides traceable, structured, and domain-compliant underlying data support for model fine-tuning, retrieval enhancement generation, or capability assessment. This original technical data may include, but is not limited to, physicochemical test reports related to insulating gases, academic papers, equipment operation manuals, international standards and specifications, safety data sheets, inspection images, and expert experience data.
[0056] The unique identifier of the original technical document is a globally unique code assigned to it. This code distinguishes documents from different sources and of different types, and remains unchanged throughout the data processing flow. This ensures that all subsequent candidate corpus fragments, question-answer pairs, or dataset samples generated based on the original technical document can be back-linked to the unique original technical document through this unique identifier, avoiding confusion between multiple documents or broken traceability. For example, when multiple corpus units come from the same paper, each corpus unit shares the same unique identifier of the original technical document; while when corpus units come from test reports and equipment manuals respectively, physical isolation is achieved through the unique identifiers of the different original technical documents.
[0057] For example, a certain sulfur hexafluoride ( The paper on alternative gases can be marked as DOC-0001, for example, a sample of perfluoroisobutyronitrile / carbon dioxide (PFOS). A breakdown test report can be marked as DOC-0002, and a manual for a gas-insulated metal-enclosed switchgear (GIS) can be marked as DOC-0003. Subsequently generated corpus units, candidate corpus fragments, and question-answer pairs can all be traced back to their corresponding original technical documents through their unique data identifiers.
[0058] A corpus unit can be understood as a basic content block obtained after structured parsing of original technical documents. Each content block contains the original text corpus from the original technical documents, serving as a standardized intermediate data carrier for subsequent semantic segmentation and question-answer pair construction. To facilitate precise management of massive corpus units throughout their entire lifecycle, each corpus unit can also have a unique unit identifier to achieve independent differentiation and globally unique addressing of different corpus units within the same document. For example, the unique unit identifier of a corpus unit originating from the original technical document DOC-0008 can be recorded as DOC-0008-part01, or as another identifier code unrelated to the unique document identifier of the original technical document.
[0059] Parsing raw technical documents into at least one corpus unit essentially involves extracting the multidimensional structural information of the raw technical documents, thereby transforming unstructured or semi-structured heterogeneous raw documents into intermediate representations with unified data standards. For example, extracting multidimensional structural information such as the main text, titles, page numbers, paragraphs, table text, image descriptions, and attachment relationships from the raw technical documents yields multiple content blocks that constitute the raw technical documents, serving as the various corpus units obtained from the parsing of the raw technical documents.
[0060] For example, a document is usually parsed into multiple corpus units. For instance, according to the document's editing format structure (text, tables, images, etc.) or content logical structure (title, abstract, etc.), it can be parsed into title units, abstract units, body paragraph units, experimental condition units, table units, image caption units, and conclusion units.
[0061] The process of parsing raw technical data into at least one corpus unit can be achieved using one or a combination of text parsing, layout recognition, table recognition, image extraction, OCR recognition, or multimodal parsing methods. In engineering implementation, methods such as PDF parsing, Word parsing, OCR, table parsing, or visual language models can be selected based on the type of raw technical data to complete the data parsing.
[0062] Based on the heterogeneity of the original technical data in terms of format, layout and content organization, and the requirements of downstream semantic segmentation and sample generation tasks for data standardization and traceability, after extracting the multi-dimensional structural information of the original technical data to obtain content blocks, standardization processing can be performed on multiple content blocks that make up the original technical data. This standardization processing includes, but is not limited to, removing obvious noise, unifying line breaks, retaining title levels, recording page numbers and paragraph numbers, identifying gas names, identifying experimental conditions and retaining source locations, forming corpus units carrying unit source information.
[0063] In the case of images as corpus units, the standardization process also includes associating image locations, that is, establishing a correspondence between the image and its page number, figure number, figure caption, adjacent paragraphs and experimental projects in the original technical documents, so that the image has a structured semantic context and traceable document anchors, thereby enabling it to participate in image question answering, image-text retrieval and multimodal evaluation dataset generation.
[0064] For example, if a certain corpus unit is an image of an experimental platform, after associating the image location, the corpus unit can be recorded as:
[0065] Image ID: IMG-0007
[0066] Source: DOC-0002
[0067] Page number: Page 6
[0068] Drawing number: Figure 3
[0069] Image caption: Mixed gas breakdown test platform
[0070] Related paragraphs: P6-02, P6-03
[0071] Based on the above records, it is possible to identify which document the image comes from, which page it corresponds to, and which text descriptions or experimental content it is related to, thereby avoiding a disconnect between the image and the text meaning.
[0072] The corpus unit carries unit source information, realizing two-level positioning at the data level and content level. This facilitates accurate location of the corpus unit within the original technical data. It includes the unique data identifier of the original technical data to which the corpus unit belongs, and the position information of the corpus unit within its original technical data. This position information is a precise description of the spatial coordinates of the corpus unit within the original data. The specific form can be dynamically determined according to the type of the original technical data and the parsing results, including one or more combinations such as page number, paragraph number, title level path, and figure number, to distinguish multiple corpus units within the same data. When multiple corpus units share the same unique data identifier, the specific content block can be accurately located through the position information.
[0073] For example, for text-based corpus units, the location information can be represented as "P12-Para3-H2.4" (page 12, paragraph 3, second-level heading 2.4); for image-based corpus units, it can be represented as "P7-Fig5" (page 7). Figure 5 This location information directly corresponds to operations such as "recording page numbers and paragraph numbers, and associating image locations" in the standardization process, ensuring that the semantic content of the corpus unit is bound to its physical / logical location in the original technical data.
[0074] The unit source information is used to meet the audit requirements of data source in subsequent steps, such as verifying whether the question-answer pair truly cites the original technical materials. As an inherent attribute of the corpus unit, the unit source information is completely transmitted to the downstream process, so that the fragment source information of the candidate corpus fragments generated in S120 and the source fragments of the question-answer pair generated in S140 can be derived or verified based on this, thereby constructing a traceable data chain.
[0075] To facilitate a better understanding of the unit source information by those skilled in the art, a parsing example is provided below. Assume the system receives an original technical document titled "X Mixed Gas Insulation Performance Test Report," with the unique document identifier assigned as DOC-0008. After processing by the parsing module, the report is split into multiple corpus units, two of which, along with their unit source information, are as follows:
[0076] (1) Corpus Unit 1
[0077] Content: "Analysis of the variation of breakdown voltage with air pressure"
[0078] Unique Data Identifier: DOC-0008
[0079] Location information: P15-Para2-H3.1 (This indicates that the corpus unit is located on page 15, paragraph 2 of the report, and belongs to the third-level heading 3.1 "Influence of air pressure on breakdown characteristics").
[0080] (2) Corpus Unit 2
[0081] Content: "Breakdown voltage-pressure curves under different mixing ratios":
[0082] Unique Data Identifier: DOC-0008
[0083] Location information: P16-Fig4 (This indicates that the image is located on page 16 of the report, and the figure number is Fig.4)
[0084] When constructing an image-text retrieval dataset, the curve can be accurately retrieved by combining "DOC-0008+P16-Fig4" and associated with its caption "Fig.4 Breakdown voltage vs. gas pressure for X mixtures at 25℃" and the experimental conditions in adjacent paragraphs, ensuring that the retrieval results simultaneously include images, text descriptions, and source credentials.
[0085] In the example above, the two corpus units belong to the same original data (shared DOC-0008), but they achieve precise content-level differentiation through differentiated location information.
[0086] For example, consider an insulation gas breakdown test report. If the report includes the following information: "Under conditions of 0.5 MPa and 25℃, for..." The average breakdown voltage of the mixed gas underwent a power frequency breakdown test and was measured to be 68.5kV. The following corpus units can then be generated:
[0087] Unique data identifier: DOC-0002;
[0088] Location information: Page number: Page 5; Paragraph number: P5-03; Content type: Main text paragraph;
[0089] The corpus unit includes the following corpus content: under conditions of 0.5 MPa and 25℃, the corpus content... A power frequency breakdown test was conducted on the mixed gas, and the average breakdown voltage was measured to be 68.5 kV.
[0090] Keywords: Breakdown test, temperature, pressure, breakdown voltage.
[0091] In this example, the corpus unit includes not only the corpus content and unit source information, but also unit keywords. These keywords reflect the core knowledge elements or key parameters of experimental facts in the field of insulating gases carried by the corpus unit. They are an explicit extraction of high-value information items such as technical entities, experimental conditions, and performance indicators in the corpus content. They can serve as the basis for subsequent label assignment of candidate corpus fragments that cite the corpus unit, enabling downstream tasks to quickly retrieve relevant corpus fragments and construct a question-answer pair label system based on keywords.
[0092] If the content of the original data is in the form of tables, then table-type corpus units can be formed; if the content is in the form of image descriptions, then image description-type corpus units can be formed. All types of corpus units retain consistent source information for subsequent segmentation, generation, and quality control.
[0093] Regarding step S120, this step essentially transforms the structured parsed corpus units into corpus segments with independent semantic integrity and knowledge atomicity, in order to address the imbalance in semantic density, information dimension, and contextual dependence of corpus units. Any corpus unit is divided into at least one candidate corpus segment, or multiple corpus units that are adjacent in position within the same original technical document are merged into one candidate corpus segment.
[0094] Experimental fact records are structured descriptions of objectively generated conditions, parameters, observed phenomena, and conclusions during insulating gas-related tests. Their core characteristics lie in verifiability and completeness of experimental elements, typically including key elements such as the test object (e.g., mixed gas X), test conditions (e.g., gas pressure, temperature, electrode spacing), test indicators (e.g., breakdown voltage, partial discharge initiation voltage), and experimental results. Unlike subjective inferences or theoretical derivations, experimental fact records directly originate from test reports, test data tables, or results analysis sections in original technical documentation.
[0095] Knowledge statements in the field of insulating gases represent a systematic discussion of non-experimental content such as the physicochemical properties, mechanisms of action, application scenarios, technical standards, or comparative analyses of insulating gases. Their core characteristics lie in conceptual completeness and logical consistency, and they typically exist in the form of definitions, explanations of principles, application descriptions, or references to standards. These knowledge statements do not rely on specific experimental scenarios but rather express general knowledge in the field, providing semantic support for generating conceptual, principle-based, or application-based question-and-answer pairs.
[0096] The structural-semantic composite unit represents a knowledge carrier that simultaneously satisfies structural boundary constraints and semantic integrity requirements. Its structural dimension indicates that the semantic segmentation of candidate corpus fragments strictly follows the title hierarchy (such as titles, paragraphs, and figure / table numbers) or predefined segmentation strategies of the original technical documents, ensuring that the fragments have clear start and end positions within the original technical documents. Its semantic dimension indicates that the content within a fragment can independently express a complete knowledge point or factual fragment, and can be accurately understood without relying on external context. The structural-semantic composite unit can be precisely traced back to the original text position of the technical documents through fragment source information, and can also participate as an independent unit in question-answering pair construction, semantic retrieval, or multimodal dataset generation.
[0097] The experimental facts in this application should be interpreted broadly, including but not limited to experimental operations conducted to verify or explore the effects of a specific technology and experimental operations conducted to measure or detect technical parameters. Those skilled in the art will understand that both are presented in the form of repeatable observational data in the technical documentation; therefore, this application uniformly uses experimental facts to summarize them without creating ambiguity.
[0098] Given the semantic structural differences between experimental facts and theoretical statements in technical data related to insulating gases, and the characteristic that a single corpus unit may simultaneously contain multiple types of heterogeneous knowledge, in the process of semantically segmenting the corpus content of each corpus unit parsed from the original technical data into at least one candidate corpus fragment, it is necessary to consider how to avoid mechanical text truncation that would damage the integrity of the experimental data, while simultaneously ensuring the semantic coherence and structural adaptability of non-experimental content. Based on this, see [reference needed]. Figure 2 The illustrative diagram illustrates a process for semantic segmentation of corpus units parsed from original technical data to obtain candidate corpus fragments. Candidate corpus fragments can be obtained through at least the following steps:
[0099] S201, when it is detected that the original technical data contains experimental facts, the experimental data describing the experimental facts in each corpus unit is identified and extracted, and the experimental data from different corpus units that belong to the same experimental fact are merged to generate a first corpus segment corresponding to the same experimental fact; different experimental facts correspond to different first corpus segments respectively.
[0100] The identification of experimental facts can be based on predefined experimental element templates, which include, but are not limited to, experimental subjects, experimental conditions, test indicators, and test results. These elements collectively describe the same experimental fact and should not be segmented into unrelated fragments. When experimental elements such as experimental conditions, gas ratios, temperature and pressure, breakdown results, decomposition products, or detection conclusions are identified in the corpus units parsed from the original technical data, structural information and semantic relationships can be comprehensively considered to determine whether adjacent content belongs to the same technical fact. Specifically, based on the experimental elements, further consideration can be given to heading level, paragraph distance, table headings, figure captions, semantic similarity, and test number to determine whether they should be merged and retained.
[0101] The purpose of merging experimental data from different corpus units that belong to the same experimental fact is to reorganize related experimental elements scattered in different paragraphs or even different chapters into complete knowledge units, and to prevent the loss of the "condition-result" correspondence due to the physical segmentation of corpus units.
[0102] For example, the following experimental facts appear consecutively in the original text of the original technical data:
[0103] Test conditions: 0.5 MPa, 25℃;
[0104] Gas ratio: =5% / 95%;
[0105] Test results: The average breakdown voltage was 68.5 kV;
[0106] Conclusion: The mixed gas has good insulation properties.
[0107] When the above content in the original technical data is divided into different corpus units, the experimental corpus content describing the same experimental fact in different corpus units is merged into a candidate corpus segment containing complete experimental conditions and results, rather than splitting it into two semantically incomplete segments.
[0108] Given multiple adjacent corpus units describing multiple different experimental facts, multiple first corpus segments are formed. Each first corpus segment aggregates experimental corpus content related to one experimental fact from multiple adjacent corpus units.
[0109] For example, in the original technical documentation, "3.2" The section on "Breakdown Characteristics of Mixed Gases" records two sets of comparative experiments consecutively:
[0110] Corpus unit DOC-0004-11: "Experimental conditions: 0.5MPa, 25℃; gas ratio: " =5% / 95%; average breakdown voltage is m kV.
[0111] Corpus unit DOC-0004-12: "Experimental conditions: 0.5MPa, 25℃; gas ratio: " =5% / 95%; the average breakdown voltage is n kV.
[0112] At this point, two independent sets of experimental facts were identified, and two first corpus fragments were generated respectively:
[0113] First corpus segment A (corresponding to) (Experimental Facts): All contents of the aggregated corpus unit DOC-0004-11 are used to form a complete first set of experimental fact records;
[0114] First corpus segment B (corresponding to) Experimental Facts): All the contents of the aggregated corpus unit DOC-0004-12 are used to form a complete second set of experimental fact records.
[0115] Through the above processing, even if multiple different experimental facts are physically adjacent in the original text and share some experimental conditions (such as the same air pressure and temperature), the system can still accurately distinguish the boundaries of each experimental fact based on the experimental element template, avoiding the incorrect merging of experimental results with different ratios.
[0116] If the original technical data is found to contain no experimental facts, the experimental corpus content is empty or does not exist, and the process can proceed directly to step S202 for semantic segmentation.
[0117] S202, for the remaining corpus content of each corpus unit excluding the experimental corpus content, the corpus is segmented according to the segmentation strategy to generate at least one second corpus fragment; the first corpus fragment and the second corpus fragment constitute each candidate corpus fragment obtained by semantic segmentation of each corpus unit; the segmentation strategy includes: structural segmentation according to the title level of each corpus unit.
[0118] The segmentation strategy refers to the segmentation method of the remaining corpus content other than the experimental corpus content, with the aim of dividing the corpus content included in the same original technical data into logically complete and semantically independent knowledge fragments.
[0119] This segmentation strategy prioritizes utilizing the inherent hierarchical structure of the original technical documents as semantic boundaries, ensuring that the generated second corpus fragments are thematically cohesive and avoiding the forced splicing of discussions belonging to different technical topics. For example, for experimental reports containing standard chapters such as "1. Introduction," "2. Materials and Methods," and "3. Results and Discussion," the content under each chapter can be independently divided into a second corpus fragment using second- or third-level headings, allowing each fragment to focus on the description of a single technical aspect.
[0120] Furthermore, based on the structural segmentation using heading levels, a segmentation method combining recursive delimiters and / or semantic length thresholds can be used to optimize the segmentation granularity when the content under a single heading level is too long or contains multiple parallel subtopics. This prevents the generated second corpus fragments from being semantically overloaded, exceeding the model processing window, or being mixed with multiple independent knowledge points due to the excessively wide coverage of a single heading. Thus, while ensuring the cohesion of the topic, the fragment length is balanced and the knowledge expression is atomized.
[0121] Alternatively, when title hierarchy information is missing or the granularity of the hierarchy is too large to support the segmentation of candidate corpus fragments, recursive delimiters and / or semantic length thresholds can be used for supplementary segmentation to compensate for the limitations of title hierarchy-driven segmentation. For example, when the data is continuous text without titles, titles are lost after scanning and recognition, or there are only first-level titles without sub-level sub-structures, recursive delimiters can identify potential semantic breakpoints based on secondary formatting features such as paragraph breaks, list symbols, and table borders. Semantic length thresholds can serve as a fallback mechanism to forcibly cut excessively long text blocks, thereby generating semantically complete and length-controllable second corpus fragments even when structural information is incomplete. This avoids the entire document being output as a single giant fragment due to structural deficiencies, which would affect the accuracy of subsequent knowledge extraction and question-answering construction.
[0122] Through the above steps, when the original technical data includes test conditions, gas ratios, temperature and pressure, breakdown results, decomposition products, or test conclusions, during semantic segmentation, adjacent content belonging to the same test fact or the same technical conclusion is identified, and they are kept in the same candidate corpus segment as much as possible, avoiding the separation of test conditions, gas ratios, test results, and conclusions.
[0123] The fragment source information carried by the candidate corpus fragment reflects the specific location information of the candidate corpus fragment within the original technical data. It can be determined by the location information of the corpus content within the candidate corpus fragment within its respective corpus unit, combined with the unit source information carried by its respective corpus unit.
[0124] When a candidate corpus fragment includes the entire content of its parent corpus unit, the position information of the content within that corpus unit does not need to be recorded or stored separately. Its fragment source information can be directly inherited from the unit source information of the corpus unit to which the candidate corpus fragment belongs. For example, when corpus unit DOC-0008-part03 is treated as a single candidate corpus fragment, its fragment source information is "DOC-0008+P12-Para2-H3.1 (Original technical document DOC-0008, page 12, paragraph 2, level 3 heading 3.1)," without requiring additional fragment-level position identifiers.
[0125] When a candidate corpus fragment contains local content of its own corpus unit, the positional information of the content within that corpus unit indicates the precise offset or scope of that local content within the corpus unit (e.g., sentence number, character start and end positions, substructure number within a paragraph, etc.). The fragment's source information includes both the unit source information and the fragment-level positional information of the corpus unit to which the candidate corpus fragment belongs. For example, when corpus unit DOC-0004-10 (unit source information is "Chapter 3 > Section 3.2 > Paragraph 2") is semantically segmented into two candidate corpus fragments, the first fragment's source information is "DOC-0004 + Chapter 3 > Section 3.2 > Paragraph 2 > Sentence 1," and the second fragment's source information is "DOC-0004 + Chapter 3 > Section 3.2 > Paragraph 2 > Sentence 2."
[0126] When candidate corpus fragments contain corpus content from different corpus units, it is necessary to aggregate the source information of all relevant corpus units, recording the unit source information of each corpus unit and the position information of the corpus content included in the candidate corpus fragment within its respective corpus unit, to form the fragment source information of the candidate corpus unit. For example:
[0127] Corpus unit DOC-0008-part05: "Experimental conditions: temperature 25℃, air pressure 0.4MPa" (location information: P15-Para1-Sent3);
[0128] Corpus unit DOC-0008-part06: "Measured breakdown voltage is 35.2kV" (Location information: P15-Para2-Sent1);
[0129] When the two are combined into a single candidate corpus fragment describing the complete experimental facts, the fragment source information of this candidate corpus fragment is as follows:
[0130] The symbols “[DOC-0008+P15-Para1-Sent3] & [DOC-0008+P15-Para2-Sent1]” are used to clearly identify that the segment is composed of the contents of two independent corpus units, each retaining its original position anchor.
[0131] This method constructs hierarchical fragment source information, enabling candidate fragments to be accurately traced back to the original technical data even after fine-grained segmentation or cross-unit merging of corpus units. This satisfies the downstream question-and-answer pair's requirement for accurate citation during generation and avoids the risk of data source chain breakage caused by segmentation operations.
[0132] For example, a certain product called "Sulfur Hexafluoride ( The document "Research on the Insulation Performance of Alternative Gases.pdf" (DOC-0004) includes corpus unit DOC-0004-10.
[0133] 3.2 Breakdown characteristics of mixed gases
[0134] Experiments show that at a gas pressure of 0.4 MPa, The insulating strength of the mixed gas is approximately that of the pure gas. a%. This gas mixture has a low global warming potential (GWP<1) and is suitable for medium-voltage switchgear.
[0135] This corpus unit can be semantically segmented into multiple candidate corpus fragments:
[0136] (1) Candidate corpus fragment 1: At a pressure of 0.4 MPa, The insulating strength of the mixed gas is approximately that of the pure gas. a%.
[0137] Fragment source information: Unique data identifier of the original technical data: DOC-0004; Location information: Chapter 3 > Section 3.2 > Paragraph 2 > Sentence 1.
[0138] (2) Candidate corpus fragment 2: The mixed gas has a low global warming potential (GWP < 1), making it suitable for medium-voltage switchgear.
[0139] Fragment source information: Unique data identifier of the original technical data: DOC-0004; Location information: Chapter 3 > Section 3.2 > Paragraph 2 > Sentence 2.
[0140] In addition, each candidate corpus segment obtained through semantic segmentation in step S120 carries not only fragment source information but also context windows and segmentation strategy information. Context windows refer to the supplementary context reserved before and after the candidate corpus segment, such as the preceding or following section of the original technical data, or the corpus content of N specified characters before and after the segment. Their function is to provide necessary context for the model when generating answers, verifying sources, or performing retrieval enhancement generation, avoiding the generation of incomplete answers based solely on isolated fragments. Segmentation strategy information refers to the formation method of the candidate corpus segments; for example, segmentation by title level, segmentation by length threshold, merging by trial number, merging by chart description, or merging by semantic similarity. Recording the segmentation strategy helps in subsequent tracking of fragment sources, explaining the reasons for segmentation, and optimizing segmentation results.
[0141] Regarding step S130, the insulating gas domain tag tree represents a structured knowledge organization system built for the field of insulating gas technology, used for multi-dimensional semantic indexing of candidate corpus fragments. It contains hierarchical tag nodes and their semantic descriptions. Each tag node has a unique identifier name and can also be associated with the tag's definition, synonym / alias set, hierarchical relationships, and semantic associations with other dimension tags. This allows the tag tree to serve as a standardized classification framework for manual review and knowledge management, as well as a structured semantic index foundation for automatic machine annotation, vector retrieval, and question-answering generation.
[0142] For example, based on the knowledge organization and model application needs in the field of insulating gases, the label tree for the insulating gas field can include at least nine main labels: gas system, equipment object, test item, performance index, decomposition product, detection method, operating status, safety and environmental protection, and application scenario. These main labels are not mutually exclusive categories but rather parallel dimensions, and further subdivided label levels can be set under each main label. For example:
[0143] Gas systems can be further subdivided into "pure gases" (such as...) , ) and "mixed gas" (such as , The mixed gas node can be further categorized according to the type of buffer gas (e.g., ...). base, Sub-labels should be set for base (air base) and typical mixing ratio ranges (such as low concentration <10%, medium concentration 10%-30%).
[0144] The equipment objects can be further subdivided into "GIS", "GIL", "circuit breaker", "current transformer", etc., and each equipment node can be further set with lower-level nodes according to dimensions such as gas chamber structure type and key voltage level;
[0145] The test items can be further subdivided into "breakdown test", "partial discharge test", "thermal stability test", "material compatibility test", etc., and each test node is associated with the standard basis (such as IEC 60060, GB / T 16927) and typical test condition templates;
[0146] The performance indicators can be further subdivided into "breakdown voltage", "partial discharge quantity", "liquefaction temperature", "GWP value" and "toxicity level", with the associated units, test methods and reasonable ranges of values for each indicator node;
[0147] Decomposition products can be further subdivided into "stable decomposition products" (such as...) CO) and "reactive decomposition products" (such as HF, CO) and "reactive decomposition products" Each product node is associated with its generation conditions, detection limits, and hazard level to equipment / human body;
[0148] The detection methods can be further subdivided into "electrical methods", "chemical methods", "spectral methods", "sensor methods", etc., with each method node related to the applicable objects, detection limits, response times and degree of standardization;
[0149] The operational status can be further divided into "newly commissioned", "normal operation", "post-failure", "retirement assessment", etc., with typical characteristic parameters and operation and maintenance handling suggestions associated with each status node;
[0150] Under the category of safety and environmental protection, it can be further subdivided into "ODP value", "GWP value", "atmospheric life", "occupational exposure limit", "waste disposal", etc., with each node being associated with regulations, standards and compliance thresholds;
[0151] Application scenarios can be further subdivided into "environmentally friendly alternatives", "high-altitude and cold regions", "high altitude", "compact design", "DC power transmission", etc., with each scenario node associated with specific technical requirements and recommended suitable gas systems.
[0152] In retrieval enhancement generation scenarios, corpus tags for candidate corpus fragments can be used for precise recall, such as recalling only " Candidate corpus fragments related to "breakdown test, GIS" are used. In model evaluation scenarios, corpus tags can be used to control the distribution of the test set and avoid test items being concentrated in one area for a long time. Alternatively, it can be used with a single alternative gas. In fine-tuning scenarios, corpus labels can be used to control the coverage of training data and improve the model's learning balance across different gas systems, equipment objects, and experimental projects.
[0153] The tag hierarchy of the main tag can be determined in one or more of the following ways: (1) Based on the preset of domain experts, form , , (1) Basic tags such as GIS, GIL, breakdown voltage, decomposition products, etc.; (2) Extract from the document title, directory and chapter name of the original technical data, such as "power frequency breakdown test" and "decomposition characteristic analysis"; (3) Identify professional terms such as "arc extinguishing", "toxicity", "greenhouse effect" and "partial discharge" based on term dictionary and keyword matching; (4) Form candidate sub-tags by semantic clustering of candidate corpus frequency bands, and then merge, split or correct them manually.
[0154] When labeling the candidate corpus fragments, the following methods can be used: (1) Based on the determination method of the pre-trained language model, that is, the pre-constructed insulating gas domain label tree and the text content of the candidate corpus fragments are used as input and fed into the domain-fine-tuned pre-trained language model. The model directly outputs the label path or label set of each corpus label that matches the semantics of the fragment based on the joint reasoning of context semantics and label tree structure; (2) Based on the method of keyword extraction and label tree matching, that is, the candidate corpus fragments are subjected to key entity recognition and technical term extraction to obtain a keyword set including gas composition, equipment name, test type, performance parameters, etc., and then the set of keywords is used to determine the label path or label set of each corpus label that matches the semantics of the fragment; The keyword set is matched precisely or fuzzily with the name, alias and related word list of each tag node in the tag tree. Based on the matching result and the hierarchical constraint relationship of the tag tree, the corpus tag of the candidate corpus fragment is determined; (3) The semantic similarity matching method based on vector retrieval is to convert each tag node in the tag tree into a semantic vector representation and construct a tag vector index in advance. At the same time, the candidate corpus fragment is encoded into a text vector. By calculating the cosine similarity or dot product similarity between the text vector and each tag node vector, the tag nodes with similarity exceeding the preset threshold are recalled as candidate tags. Then, the hierarchical logic of the tag tree is combined to perform deduplication and verification to determine the corpus tag of the candidate corpus fragment.
[0155] It is understood that the implementation method for determining the corpus tags of candidate corpus segments given here is an exemplary illustration. Those skilled in the art can also choose any other applicable technical means that can achieve the determination of corpus tags according to actual needs. This application does not limit this.
[0156] The same candidate corpus fragment can have multiple corpus tags simultaneously. For example, the fragment content is " When reporting the breakdown test results of mixed gases in GIS equipment, the following labels may be used simultaneously:
[0157] Gas system - mixed gas -
[0158] Device Objects - GIS
[0159] Test item - breakdown test
[0160] Performance Specifications - Breakdown Voltage
[0161] Application Scenarios - Environmentally Friendly Alternative Gases
[0162] For step S140, the current generation task is a configuration unit defined by the specific capabilities of the insulating gas domain model to construct the target or application scenario, which is used to guide the automated production of question-answer pairs, including but not limited to tasks such as model training, model fine-tuning, retrieval enhancement, or model evaluation.
[0163] The label range of the current generation task is determined based on the core knowledge dimensions and capability boundaries corresponding to the application requirements of the insulating gas domain model. It reflects the data requirements of the subset of insulating gas domain knowledge that the generation task needs to cover in order to achieve the expected model effect.
[0164] In this step, using the corpus tags of candidate corpus fragments, we first filter out each candidate corpus fragment whose tags match the specified tag range. This allows us to accurately locate knowledge materials semantically relevant to the current generation task from the entire candidate corpus fragment set, eliminating interference from irrelevant or low-relevance fragments. Furthermore, each candidate corpus fragment whose tags match the specified tag range serves as the factual basis for question-answer pair generation. The structured knowledge statements and technical elements carried within the candidate corpus fragments are used as the core elements to construct question-answer pairs, ensuring that the question-answer pairs strictly adhere to the authentic content of the original technical data in terms of question relevance and answer accuracy, avoiding model illusions or knowledge deviations.
[0165] After filtering out candidate corpus segments whose tags match the specified tag range, the source information of the candidate corpus segment can be used to locate the original technical data. The integrity and corpus tags of the candidate corpus segment can be verified based on the original text in the original technical data to ensure the construction quality of the question-answer pair.
[0166] In the process of constructing question-answer pairs, in order to ensure that the generated question-answer pairs are precisely matched with the application requirements of the model in terms of formal specifications, difficulty distribution and expression style, generation constraints of the current generation task are set so that the question-answer pairs can be directly adapted to the application requirements of the model and can be put into use without additional manual cleaning.
[0167] For example, the generation constraints may include at least a question type template and output content constraints. The output content constraints are used to define the conversational structure of the question-answer pair and / or the content expression style of the sample answers. The question type template represents a pre-defined structured instruction that instructs the question generation model to generate specific types of questions, which include at least one or more of the following: numerical query, parameter comparison, causal reasoning, process sequence, and definition explanation. The structured instructions include fixed sentence templates and / or example guidance templates.
[0168] The output content constraints may include at least one of the following parameters:
[0169] Application adaptation parameters are used to characterize the target task type and interaction object characteristics of the question-answering service; the application adaptation parameters include at least one of the expert role type simulated when generating the question-answering pair, the target audience of the question-answering pair, and the model application requirements adapted to the question-answering pair.
[0170] Attribute parameters are used to characterize the expressive depth and structural features of sample answers in a question-and-answer pair. The attribute parameters include at least one of the following: conversation rounds, question difficulty level of the question-and-answer pair, answer length, and whether a source is cited. The conversation rounds represent the number of dialogue interactions in the question-and-answer pair, the question difficulty level represents the level of professional knowledge required for the answer, the answer length represents the text length level of the answer, and whether a source is cited indicates whether the sample answer contains the source identifier of the candidate corpus segment it references.
[0171] The exported parameters are used to characterize the output specifications of the question-answer pair dataset, including at least one of the number of generated pairs and the exported format; wherein, the number of generated pairs represents the number of question-answer pair samples constructed based on the same candidate corpus fragment and / or the total number of question-answer pair samples corresponding to the current generation task, and the exported format represents the file encoding structure of the question-answer pairs, including but not limited to JSONL, CSV, question-answer pairs, and evaluation set formats.
[0172] The question template is flexibly configured based on the current generated task, and may include at least one of the following:
[0173] In the case of fine-tuning the current generated task instruction model, the question template is configured as at least one of the following: an insulating gas knowledge question and answer template, an experimental condition explanation and reasoning template, an instruction response template, or a multi-round expert dialogue template; wherein, the instruction response template is used to define specific task instructions and compliance response outputs in the field of insulating gases;
[0174] When the current generation task indicates that the model retrieves enhanced generation requirements, the question template is configured as a source-tracing question-answering template. The source-tracing question-answering template includes question-answer pairs and a mapping relationship between the question-answer pairs and the candidate corpus fragments actually referenced by the generated question-answer pairs.
[0175] In the case of the current task instruction model evaluation requirements, the question type template is configured as at least one of the following: judgment question template, single choice question template, multiple choice question template, short answer question template, or open-ended question template, and the question type template is associated with a reference answer template and a scoring point template that match the question type.
[0176] Based on this, see Figure 3 The illustrative diagram illustrates a process for constructing multiple question-answer pairs based on selected candidate corpus fragments. The process of constructing multiple question-answer pairs based on selected candidate corpus fragments may include the following steps:
[0177] S301, the selected candidate corpus fragments, the output content constraints, and the question type template input question generation model are used to generate various sample questions that match the question type template;
[0178] The question generation model is used to identify questionable knowledge points from candidate corpus fragments and transform them into natural language questions that conform to the specified question type of the question template, under the structured guidance of the question type template.
[0179] The question generation model can be implemented using an open-source large language model fine-tuned with question-and-answer data from the insulating gas domain, or a general commercial large language model.
[0180] S302, Based on the candidate corpus fragments actually referenced by the generated sample question, verify the answerability of the sample question;
[0181] This step verifies the answerability of sample questions to filter those with clear and semantically consistent answers. The answerability of a sample question reflects whether the candidate corpus fragments used to generate the sample question contain clear, complete, and unambiguous answer support information for that sample question.
[0182] The answerability of the sample question can be verified by inputting the sample question and the candidate corpus fragments actually referenced in generating the sample question into another verification model to determine the implication relationship, or by detecting whether the fragments contain the key entities and attribute values involved in the question based on a rule engine.
[0183] If it is determined that the answer to the sample question can be directly or unambiguously inferred from the candidate corpus fragments actually referenced in generating the sample question, then the sample question is determined to be answerable, the answerability check passes, and the process proceeds to the sample answer generation step S303; otherwise, if the answer to the sample question cannot be directly or unambiguously inferred from the candidate corpus fragments actually referenced in generating the sample question, then the sample question is determined to be unanswerable, the answerability check fails, and the sample question can be discarded directly.
[0184] S303, input the sample questions that have passed the answerability verification, the selected candidate corpus fragments, and the output content constraints into the answer generation model to generate a sample answer for each sample question;
[0185] The answer generation model is used to generate a response to the sample question based on the factual content in the candidate corpus fragments, under the constraints of the output content, thus avoiding the model from introducing external knowledge or making arbitrary decisions. This answer generation model can be implemented using a language model that is the same as or different from the question generation model, but has been fine-tuned specifically for answer generation.
[0186] For example, the question generation model and the answer generation model can be different fine-tuned versions of the same base model or different inference sessions of the same language model.
[0187] S304, verify whether the sample answer corresponding to the sample question satisfies the output content constraint, and combine the sample answer that satisfies the output content constraint and its sample question into a question-answer pair.
[0188] The sample answers output by the answer generation model may contain generation biases. This step further verifies whether the sample answers corresponding to the sample questions meet the output content constraints. This verification process can be automatically implemented through regular expression matching, structured parsing verification, or constraint compliance scoring models. High-quality question-answer pairs that pass the verification are included in the output set, thereby ensuring the consistency of the generated data format and the compliance of the content.
[0189] Through the above steps, question-answer pairs that conform to the current generation task label range and generation constraints and are highly consistent in question type structure and content expression style are generated in batches. This realizes the automated construction and verification of candidate corpus fragments into high-quality question-answer pairs, and provides standardized data for application needs such as fine-tuning training of insulating gas models, construction of retrieval enhancement generation knowledge base, and model capability evaluation.
[0190] For the generated question-and-answer pairs, post-processing quality control steps can be performed, including but not limited to deduplication, cleaning, quality scoring, regeneration of low-scoring pairs, and manual review and labeling. When the quality score falls below a preset threshold, a regeneration mechanism is automatically triggered or the pair is marked for manual review to ensure the reliability and usability of the final exported sample. The post-processing quality control steps can include a two-stage process:
[0191] Basic format cleaning is used to eliminate output noise, including but not limited to removing garbled characters, merging duplicate spaces, removing meaningless symbols (such as isolated punctuation marks and invisible control characters), and clearing residual traces of prompt word templates (such as unreplaced placeholders and instruction prefixes).
[0192] Semantic verification and scoring processing include, but are not limited to, duplicate sample detection, insulating gas terminology verification, and quality scoring. Duplicate sample detection identifies and marks highly similar or completely identical question-and-answer pairs to prevent data redundancy from causing model overfitting. Insulating gas terminology verification checks the spelling accuracy and usage standardization of key terms (such as gas names, equipment components, and fault phenomena) against a domain terminology dictionary, correcting homophones, misspellings, or non-standard abbreviations. The quality score represents the quality rating of the question-and-answer pair.
[0193] Based on the quality score of the question-answer pairs, the generated question-answer pairs can be categorized and processed according to different paths. For example, when the quality score of a question-answer pair is higher than a first score threshold, the question-answer pair is retained; when the quality score of a question-answer pair is lower than the first score threshold but higher than a second score threshold, the sample answer of the question-answer pair is regenerated based on the source fragment associated with the question-answer pair; when the quality score of a question-answer pair is lower than a third score threshold, the question-answer pair is marked as requiring manual review or is directly discarded and not included in the construction of the target dataset.
[0194] For step S150, this step is used to filter and export the dataset that meets the data requirements of the current generation task from the generated question-answer pairs after the question-answer pairs have been constructed.
[0195] For example, the multiple question-answer pairs already constructed under the current generation task can be directly organized into a set as the target dataset. Alternatively, with the goal of sample balance, a subset of question-answer pairs that meet the preset threshold in each dimension can be selected from the multiple question-answer pairs already constructed under the current generation task to form a set as the target dataset.
[0196] When question-answer pairs constructed under different generation tasks are stored in a mixed manner, since different question-answer pairs adopt a unified generation method and index management, the label range and generation constraint information under the model application requirements of the target dataset can be constructed as needed. A subset of question-answer pairs that meet the model application requirements can be selected from the mixed-stored question-answer pairs, and the target dataset can be constructed based on the selected subset of question-answer pairs.
[0197] To further improve the reliability of the target dataset and the accuracy of its domain knowledge, before constructing the target dataset corresponding to the current generation task, data quality verification can be performed on the multiple question-answer pairs already constructed under the current generation task, namely:
[0198] Perform data quality verification for each question-and-answer pair; the data verification includes at least: verifying whether the question-and-answer pair can be supported by the source fragment associated with the question-and-answer pair; and / or verifying whether the question-and-answer pair mixes conclusions from different gas systems, different equipment types, or different test conditions; and integrating the question-and-answer pairs that have passed the data quality verification to obtain the target dataset.
[0199] Each question-and-answer pair is associated with a source fragment; the source fragment includes candidate corpus fragments that were actually referenced when the question-and-answer pair was generated.
[0200] Verifying whether a question-and-answer pair can be supported by the associated source fragment essentially involves checking whether all factual statements, numerical parameters, and logical inferences in the sample answer can find a clear and unique corresponding basis in the source fragment. This confirms that the question-and-answer pair has not introduced external knowledge, has not produced content illusions, or has not exceeded the information boundaries of the original corpus. This verification process can be achieved through methods such as cross-comparison of entity attributes based on structured tags, judgment of implication relationships based on natural language reasoning models, or verification of the legitimacy of reasoning paths based on domain knowledge graphs.
[0201] The situation described as mixing conclusions from different gas systems, different equipment types, or different test conditions essentially refers to the incorrect association or merging of experimental conclusions belonging to different experimental facts. Since a complete experimental fact is typically uniquely determined by three core parameters—gas system, equipment type, and test conditions—the mixed use of conclusions from different experimental facts can be automatically detected by separately verifying cross-boundary referencing scenarios across these three dimensions.
[0202] For example, fact A: Gas, GIS circuit breaker, -40℃ environment → breakdown voltage is X;
[0203] Fact B: Mixed gas, disconnecting switch, 20℃ environment → breakdown voltage is Y;
[0204] If the sample answer treats the conclusion of fact B as the conclusion of fact A, or mixes the conclusions of fact A and fact B, that is, mixes the conclusions that constitute different experimental facts.
[0205] Therefore, the question-and-answer pair can be verified in any of the following ways to determine whether conclusions from different gas systems, different equipment types, or different test conditions are mixed:
[0206] (1) Verification based on structured tags involves converting unstructured corpora and answers into structured metadata for comparison:
[0207] Step 1: Obtain the corpus tags for the source segment;
[0208] Step 2: Perform named entity and technical term recognition on the sample answers in the question-answer pairs, and extract keywords such as gas names, equipment models, and test parameter values.
[0209] Step 3: Compare the keywords extracted from the answer with the tags of the source segment one by one. If any of the following situations occur, it is judged as "conclusion misuse":
[0210] The gas system labels appearing in the answer are not the same as the gas system labels in the source segment;
[0211] The test conditions cited in the answer exceed the range stated in the source fragment label (e.g., the source is labeled "-40℃", but the answer contains "breakdown voltage at -30℃").
[0212] The answer contained the names of two or more different gas systems without clearly distinguishing the applicable scenarios.
[0213] (2) Validation based on natural language reasoning model
[0214] Construct premise-hypothesis pairs as input for natural language reasoning tasks: use source fragments as premises and sample answers as hypotheses;
[0215] Using a natural language reasoning model fine-tuned with an insulating gas domain corpus, we analyze the relationship between premises and assumptions in the input premise-hypothesis pair and output the corresponding prediction relationship.
[0216] If the predicted relationship output by the model contradicts the premise and assumption, it is determined that there is a case of misuse of conclusions.
[0217] (3) Validation based on knowledge graph
[0218] The gas system, equipment type, and test conditions in the source fragment are mapped to entity nodes and relation edges in the knowledge graph.
[0219] Map the key conclusions in the sample answers to target nodes in the graph.
[0220] Verify whether a valid reasoning path exists between the source node corresponding to the source fragment and the target node corresponding to the sample answer. If no valid path exists, or if the path passes through intermediate nodes that do not belong to the gas system / equipment type, then a case of mixed conclusions has been identified.
[0221] The above is merely an exemplary method for verifying mixed experimental conclusions. Those skilled in the art can flexibly choose other suitable methods to implement this method according to actual needs.
[0222] Furthermore, when using the dataset generation method described in this embodiment, to address the computational resource pressure and long processing time risks associated with large-scale candidate corpus processing, and to support interruption recovery, parallel scheduling, and fine-grained fault location, the overall data processing flow can be broken down into a series of subtasks: batch parsing, batch segmentation, batch question generation, batch answer generation, batch quality assessment, and batch export. Each subtask has states such as pending execution, in progress, partially successful, successful, failed, canceled, and expired, and records progress, error reasons, output location, and associated dataset version, facilitating real-time monitoring of task flow status and breakpoint recovery.
[0223] In this embodiment, a structured generation method from raw technical data to a target dataset is proposed. The raw technical data is parsed into corpus units with independent semantics and carrying unit source information. The semantic segments of these corpus units are then processed into candidate corpus fragments carrying specific knowledge statements or experimental facts. This effectively solves the semantic fragmentation problem caused by the mixing of charts, formulas, parameters, and text in insulating gas field literature, ensuring the integrity and traceability of the smallest knowledge granularity. Furthermore, an insulating gas field tag tree is introduced based on domain knowledge organization and application requirements. This maps unstructured corpus into standardized tag nodes, and dynamically filters candidate corpus fragments based on the tag range and generation constraints of the current generation task. The filtered candidate corpus fragments are then used as factual basis to construct question-and-answer pairs, ensuring that the constructed question-and-answer pairs follow entity knowledge or experimental facts in the insulating gas field. This significantly reduces the interference of noisy data and improves the domain professionalism and accuracy of the dataset.
[0224] Moreover, since the parsing and labeling of corpus fragments are independent of specific generation tasks, the same basic corpus can be quickly adapted to the dataset construction needs of different downstream tasks such as insulating gas composition analysis, fault diagnosis, and material selection by switching the label range and generation constraints. This avoids the high cost of repeatedly parsing data for each new task, improves the flexibility and reusability of dataset construction, and greatly enhances the efficiency of iterative development of domain models.
[0225] In addition, each question-answer pair in the final target dataset has a clear index and location information of the original data, which facilitates subsequent manual review and quality assessment, and also provides underlying data support for knowledge tracing during model inference, thereby enhancing the interpretability and credibility of the dataset.
[0226] In some embodiments, based on the aforementioned output content constraints, the output content constraints may include one or more of the following: the type of expert role simulated when generating the question-and-answer pair, the target audience of the question-and-answer pair, the model application requirements adapted to the question-and-answer pair, the number of conversation rounds, the difficulty level of the questions in the question-and-answer pair, the length of the answer, and whether or not a source is cited.
[0227] Under the constraints of this output content, in order to ensure that the constructed question-answer pairs are traceable, filterable, and adaptable to downstream tasks in terms of multi-dimensional attributes, each question-answer pair constructed under the current generation task can be configured with metadata. The metadata includes at least attribute fields and application adaptation fields. The values of the fields in the metadata are determined based on the generation constraints of the current generation task. The attribute fields include at least one of the following: sample label, question type, difficulty level, and conversation round. The sample label is determined based on the corpus labels of the candidate corpus fragments actually referenced when generating the question-answer pair. The application adaptation fields include at least one of the following: the type of expert role simulated when generating the question-answer pair, the target audience of the question-answer pair, and the model application requirements adapted to the question-answer pair.
[0228] Based on the metadata of the question-answer pairs, after constructing multiple question-answer pairs according to the selected candidate corpus fragments, considering that the distribution of the question-answer pairs generated in the initial batch may have long-tail sparsity or class imbalance problems in various statistical dimensions, in order to ensure the sufficiency of coverage and training effectiveness of the target dataset in key knowledge dimensions and task adaptation dimensions, this embodiment introduces a sample balancing process.
[0229] See Figure 4 The flowchart illustrating an example of a question-answer pair-based targeted supplementation step based on sample balancing may further include the following steps before constructing the target dataset:
[0230] S401, based on the metadata of each of the multiple question-answer pairs, determine the number of question-answer pairs of different categories under the target statistical dimension; the target statistical dimension is composed of at least one field in the metadata;
[0231] For example, if the target statistical dimension is the sample label of the question-answer pair, then the number of question-answer pairs under each sample label is counted; if the target statistical dimension is the sample label + question type, then for each sample label, the number of question-answer pairs under that sample label and for different question types is counted; if the target statistical dimension is the question type, then the number of question-answer pairs under each question type is counted.
[0232] In addition, the target statistical dimension can also be a single field or any combination thereof, such as difficulty level, expert role type, target audience, model application requirements, etc., which can be set according to the fine-tuning target of the current generated task, the evaluation scenario, or the priority of domain knowledge coverage.
[0233] S402, based on the number of question-answer pairs of different categories under the target statistical dimension, identify the target category that needs to be supplemented with question-answer pairs, and supplement question-answer pairs that meet the target category based on the selected candidate corpus fragments.
[0234] The target category for supplementing question-answer pairs typically refers to the category where, under the current target statistical dimension, the number or proportion of question-answer pairs is below a corresponding threshold, exhibiting a sparse distribution. For example, when the target statistical dimension is "sample labels" and the preset threshold is 50, if " If there are only 10 question-answer pairs under the "Decomposition Products" category, then that category is identified as the target category; for example, when the target statistical dimension is "sample label + question type", if " If there are only 3 question-answer pairs in the "mixed gas + multiple choice" combination category, while other combination categories have more than 50, then this combination category is also identified as the target category.
[0235] For the identified target categories, a targeted generation strategy is employed to generate supplementary question-answer pairs that accurately fill distribution gaps and avoid unnecessary and redundant generation. For example:
[0236] When the target category is "sample label = When referring to "decomposition products", targeted generation indicates that the corpus retrieval scope is forcibly specified in the prompt word Prompt. Decompose candidate corpus fragments related to the decomposed products and inject higher-order inference instruction constraints;
[0237] When the target category is "Question Type = Multiple Choice", targeted generation means injecting a multiple choice format template and distractor generation instructions into the Prompt.
[0238] By using the aforementioned targeted question-and-answer pair approach, the final target dataset achieves class balance across the target statistical dimensions, providing a standardized sample base with reasonable distribution and sufficient coverage for downstream model fine-tuning or evaluation.
[0239] In some embodiments, the target dataset includes a data file and a manifest file; the data file is used to record the exported question-and-answer pairs; the manifest file is used to record the metadata of the exported question-and-answer pairs, thereby forming a standardized dataset encapsulation structure in which the data content and descriptive information are physically separated but logically related.
[0240] Based on this, see Figure 5 The illustrative diagram illustrates a process for constructing a target dataset. In the process of constructing the target dataset corresponding to the current generation task based on multiple question-answer pairs already constructed under the current generation task, or in the process of performing data quality checks on the multiple question-answer pairs already constructed under the current generation task as described in the foregoing embodiments, integrating the question-answer pairs that have passed the data quality check to obtain the target dataset, or in the process of constructing the target dataset after targeted supplementation of the generated question-answer pairs based on sample balancing, the target dataset can be constructed in the following manner:
[0241] S501, from the multiple question-answer pairs already constructed under the current generation task, determine the subset of question-answer pairs to be exported according to the application requirements of the model indicated by the current generation task;
[0242] The process of determining the subset of question-answer pairs to be exported is essentially based on the metadata of the question-answer pairs, selecting pairs that meet the value criteria of at least one field in the metadata. For example, all question-answer pairs already constructed under the current generation task can be selected as the pairs to be exported; alternatively, only those with "Expert Role Type = R&D Engineer" and "Sample Tag = The question-and-answer pair subset of "decomposition products" serves as the question-and-answer pair subset to be derived for fine-tuning the high-order reasoning capability of the large model for fault diagnosis of insulating gases.
[0243] When considering directional export for balanced samples, to avoid performance degradation due to skewed sample distribution during model training or evaluation, the number of question-answer pairs exported under each group dimension is determined based on the user-input configuration file. This configuration file includes multiple group dimensions, determined based on the application requirements of the model indicated by the current generation task. These dimensions are subsets of the metadata fields of the question-answer pairs, including but not limited to sample labels, question type, difficulty, gas system, device object, source fragment, and quality score. If the number of question-answer pairs under a certain group dimension is insufficient, the number of exported pairs under that group dimension can be automatically reduced, an automatic directional supplementation of question-answer pairs can be triggered, or a manual supplementation prompt can be provided.
[0244] S502, the question-answer pair subset is exported as a data file matching the application requirements of the model using an export template; the export template is selected according to the application requirements of the model indicated by the current generation task.
[0245] The exported template represents a predefined set of formatting rules for mapping question-answer pairs and their metadata to the data structures and field specifications required for a specific downstream task.
[0246] When the application requirements of the model are used for fine-tuning, the data file format is a question-and-answer pair format or an instruction-response format; when the application requirements of the model are used for dialogue model training, the data file format is a multi-turn message format; the data file format is a retrieval-enhanced question-and-answer pair format containing sample questions, sample answers, associated source fragments, and gas system labels; when the application requirements of the model are used for evaluation, the data file format is a test set format containing sample questions, sample answers, scoring rules, and source fragments.
[0247] S503, Generate a manifest file associated with the data file; the manifest file is used to record the metadata of each question-answer pair in the question-answer pair subset.
[0248] Through the above steps, the target dataset is constructed with precise alignment between content and application requirements, and structural matching with the exported template. This allows the final output target dataset to be directly loaded into the corresponding model training or evaluation pipeline, while retaining all traceable attribute description information.
[0249] To enable those skilled in the art to better understand the dataset generation method for insulating gas domain models provided in this embodiment, the following explanation will use five typical application scenarios as examples: insulating gas document-driven generation, topic-driven distillation, image question-and-answer generation, operation and maintenance multi-turn dialogue generation, and evaluation dataset generation.
[0250] Example 1: Dataset Generation Process Driven by Documents on Insulating Gases
[0251] After a user creates a dataset project, the system receives standard procedures, test reports, papers and patents, equipment manuals, operation and maintenance records, test reports or maintenance logs, and assigns a unique document identifier to each original technical document. The system extracts the main text, titles, page numbers, paragraphs, table text, image descriptions, and attachment relationships of each original technical document to form corpus units.
[0252] The system performs standardization processing on the corpus units, including removing obvious noise, standardizing line breaks, preserving heading levels, recording page numbers and paragraph numbers, associating image locations, identifying gas names, identifying experimental conditions, and saving the original source. The standardized corpus units are not directly used as training samples, but rather as the foundational material for subsequent semantic segmentation and sample generation.
[0253] Structural segmentation is prioritized based on the document heading hierarchy of the original technical documents. When the heading hierarchy is insufficient, recursive delimiters and semantic length thresholds are used to supplement the segmentation. When the text contains experimental conditions, gas ratios, temperature and pressure, breakdown results, decomposition products, or detection conclusions, adjacent structural boundaries are preserved to avoid fragmenting the same experimental facts. The segmentation results form candidate corpus fragments, each of which preserves the original technical document referenced, page number, paragraph number, context window, and segmentation strategy information. Among these, the original technical document, page number, and paragraph number constitute the fragment source information.
[0254] An insulating gas domain tag tree is generated based on the directory title, keywords, fragment semantics, and manual correction results, and a corpus tag is assigned to each candidate corpus fragment. This insulating gas domain tag tree can include nine main tags: gas system, equipment object, test item, performance index, decomposition product, detection method, operating status, safety and environmental protection, and application scenario. Further sub-tag levels and difficulty markers can be set under the main tags.
[0255] The question generation model reads candidate corpus fragments, corpus tags, and output content constraints, and generates sample questions according to question type templates. For fine-tuning, the system generates knowledge questions and answers about insulating gases, explanations of experimental conditions, command responses, or multi-round expert dialogues. For retrieval enhancement generation, the system retains the mapping relationship between questions, answers, and source fragments. For model evaluation, the system generates true / false questions, single-choice questions, multiple-choice questions, short-answer questions, or open-ended questions, and simultaneously generates reference answers and scoring points. The output content constraints are a set of configurations that control the sample generation method, determining what data is generated, its intended use, the question type, the role of the respondent, and the final export format.
[0256] Output constraints may include parameters such as question type, purpose, role, audience, difficulty, number of rounds, answer length, whether to cite sources, tag range, number of outputs, and export format. For example:
[0257] Question types: Question and answer, multi-turn dialogue, true / false, multiple choice, picture-based questions and answers.
[0258] Applications: Fine-tuning, retrieval enhancement generation, model evaluation
[0259] Roles: Expert, Engineer, Maintenance Personnel, Instructor
[0260] Target audience: Researchers, system administrators, students, reviewers
[0261] Difficulty: Basic, Intermediate, Professional
[0262] Rounds: Single round or multiple rounds
[0263] Answer length: concise, standard, detailed
[0264] Source cited: Yes / No
[0265] Label scope: Specified gas system or test item
[0266] Quantity: Number of samples generated
[0267] Export formats: JSONL, CSV, question-answer pairs, evaluation set format
[0268] The answer generation model generates answers based on sample questions, candidate corpus fragments, and output content constraints. Before saving the question-answer pair, the system checks whether the answer can be supported by the candidate corpus fragments referenced in the question-answer pair generation process, and must not mix conclusions from different gas systems, equipment types, or experimental conditions.
[0269] Example 2: Topic-Driven Data Distillation Process
[0270] After obtaining candidate corpus fragments corresponding to each original technical document, the system first generates an insulating gas domain tag tree after the user inputs the insulating gas topic, target audience, number of samples, question type ratio, difficulty range, and output content constraints of the target gas system. Then, it generates a set of sample questions around the insulating gas domain tag tree. Subsequently, it calls the large language model to generate sample answers, explanations, and optional extended questions for each sample question.
[0271] In this process, the system can count the number of samples (i.e., question-answer pairs) based on one or a combination of gas systems, equipment objects, test items, question types, and difficulty levels. When the sample size for a certain statistical dimension is insufficient, sample questions and their answers for that statistical dimension are added first. When the sample ratio for a certain question type is too high, the generation of that question type is paused and switched to other question types. When the quality score of a question-answer pair is lower than a specified score threshold, the system triggers the regeneration of question-answer pairs or marks them for manual review.
[0272] Example 3: Image Question Answering Dataset Generation Process
[0273] After obtaining candidate corpus fragments of image types corresponding to each original technical document, the candidate corpus fragments of that image type retain the associated image location. That is, the system establishes a correspondence between the image and its page number, figure number, figure caption, adjacent paragraphs and test items in the original technical document, so that the image can participate in image question answering, image and text retrieval and multimodal evaluation dataset generation.
[0274] For images extracted from equipment photos, gas chamber structure diagrams, test platform diagrams, spectra, waveform diagrams, decomposition product charts, or PDFs, the visual model analyzes the image content. The question generation model and the answer generation model combine the image and its retained related image locations to generate sample questions and answers. The unique image identifier, sample question, sample answer, original technical data, test item, and corpus tags are written into the same sample record.
[0275] Example 4: Process for Generating Multi-Turn Dialogue Datasets for Operation and Maintenance
[0276] After acquiring candidate corpus fragments corresponding to each original technical document, the system receives the expert role type simulated when generating question-and-answer pairs, the model application requirements adapted to the question-and-answer pairs, the target conversation rounds, and dialogue constraints configured by the user. It then organizes the dialogue context based on multiple candidate corpus fragments under the same corpus tag. After each round of dialogue is generated, the system checks whether the responses are repetitive, deviate from the expert role type, exceed the original text of the source document, confuse fault phenomena with detection conclusions, and meet the target format. Rounds that fail the checks are regenerated or marked for review.
[0277] Example 5: Process for generating an insulating gas assessment dataset.
[0278] The system generates true / false, single-choice, multiple-choice, short-answer, and open-ended questions based on candidate corpus fragments. For each question, it saves the reference answer, scoring rules, question type, label, gas system, experimental conditions, and source. After the evaluated model submits its answer, the system can automatically score using the teacher's model, or anonymously display the answers from different models for blind testing by human reviewers.
[0279] This invention can be implemented as a software process in a browser application, desktop application, or server application. One optional implementation uses a front-end interface to receive project configurations, domain tag corrections, and manual review; a back-end service performs parsing, segmentation, generation, quality control, and export; and a database or file system stores projects, fragments, tags, issues, samples, quality scores, and task statuses.
[0280] For example, in one processing chain, the user uploads an insulating gas test report and equipment information, and the system parses it to obtain corpus units; the system performs semantic segmentation on each parsed corpus unit to generate candidate corpus fragments; the system configures corpus tags for the candidate corpus fragments, and generates question-answer pairs such as question-answer, dialogue, image question-answer, and assessment questions based on the selected candidate corpus fragments; after filtering low-quality samples, the system exports JSONL data files and manifest files in a balanced manner according to sample tags and question types.
[0281] Based on the above, this embodiment aims to generate multi-type datasets in the field of insulating gases. It unifies data extraction, semantic segmentation, domain label tree construction, sample generation, quality control, and sample export into a traceable automated process. It can fine-tune, enhance retrieval generation, and evaluate models for the field of insulating gases, generating high-quality datasets with consistent structure, controllable labels, traceable sources, and exportable formats.
[0282] Corresponding to the aforementioned dataset generation method for models in the insulating gas domain, see [link to relevant documentation]. Figure 6 The illustrated block diagram represents a dataset generation apparatus for an insulating gas domain model. This embodiment also provides a dataset generation apparatus for an insulating gas domain model, the apparatus comprising:
[0283] The data parsing module 610 is configured to parse each original technical document related to the field of insulating gases into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique identifier of the original technical document and the position information of the corpus unit in the original technical document.
[0284] The corpus segmentation module 620 is configured to semantically segment the corpus content of each corpus unit parsed from the original technical data into at least one candidate corpus segment; any candidate corpus segment represents a structure-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gas; the candidate corpus segment carries segment source information; the segment source information is determined based on the corpus unit to which the corpus content in the candidate corpus segment belongs and the unit source information it carries;
[0285] The corpus tag determination module 630 is configured to determine the corpus tag for each candidate corpus segment; the corpus tag includes at least one tag node in an insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize a knowledge keyword or technical topic in the insulating gas domain.
[0286] The question-answer pair construction module 640 is configured to filter out candidate corpus fragments whose corpus tags match the label range based on the label range of the current generation task, and construct multiple question-answer pairs based on the generation constraints of the current generation task and the filtered candidate corpus fragments; the current generation task is determined based on the application requirements of the model.
[0287] The dataset export module 650 is configured to construct the target dataset corresponding to the current generation task based on multiple question-answer pairs already constructed under the current generation task.
[0288] In some embodiments, the corpus segmentation module is specifically used for:
[0289] When the original technical data is found to contain experimental facts, the experimental data describing the experimental facts in each corpus unit is identified and extracted, and the experimental data from different corpus units that belong to the same experimental fact are merged to generate the first corpus segment corresponding to the same experimental fact; different experimental facts correspond to different first corpus segments.
[0290] For the remaining corpus content of each corpus unit excluding the experimental corpus content, the corpus is segmented according to a partitioning strategy to generate at least one second corpus fragment; the first corpus fragment and the second corpus fragment constitute each candidate corpus fragment obtained by semantic segmentation of each corpus unit; the partitioning strategy includes: structural segmentation according to the title level of each corpus unit.
[0291] In some embodiments, the question-answer pair is associated with a source fragment; the source fragment includes candidate corpus fragments that are actually referenced in generating the question-answer pair; the dataset export module is specifically used for:
[0292] For each question-and-answer pair, data validation is performed; the data validation includes at least: verifying whether the question-and-answer pair can be supported by the source fragment associated with the question-and-answer pair; and / or verifying whether the question-and-answer pair mixes conclusions from different gas systems, different equipment types, or different test conditions;
[0293] The target dataset is obtained by integrating the question-answer pairs that have passed the data verification.
[0294] In some embodiments, the question-answer pair is configured with metadata; the metadata includes at least attribute fields and application adaptation fields; the values of the fields in the metadata are determined based on the generation constraints; the attribute fields include at least one of sample tags, question type, difficulty level, and conversation rounds; the sample tags are determined based on the corpus tags of the candidate corpus fragments actually referenced in generating the question-answer pair; the application adaptation fields include at least one of the expert role type simulated when generating the question-answer pair, the target audience of the question-answer pair, and the model application requirements adapted to the question-answer pair.
[0295] After constructing multiple question-answer pairs based on the selected candidate corpus fragments, the device further includes:
[0296] Based on the metadata of each of the multiple question-answer pairs, the number of question-answer pairs of different categories under the target statistical dimension is determined; the target statistical dimension is composed of at least one field in the metadata.
[0297] Based on the number of question-answer pairs in different categories under the target statistical dimension, the target categories that need to be supplemented with question-answer pairs are identified, and question-answer pairs that meet the target categories are supplemented in a targeted manner based on the selected candidate corpus fragments.
[0298] In some embodiments, the generation constraints include at least a question type template and output content constraints; the output content constraints are used to limit the conversation structure of the question-answer pair and / or the content expression style of the sample answers; the question-answer pair construction module is specifically used for:
[0299] The selected candidate corpus fragments, the output content constraints, and the question type template are used to input the question generation model to generate various sample questions that match the question type template.
[0300] The answerability of the sample question is verified based on the candidate corpus fragments actually referenced in generating the sample question.
[0301] The answer generation model is jointly inputted with each sample question that passes the answerability check, the selected candidate corpus fragments, and the output content constraints to generate a sample answer for each sample question.
[0302] Verify whether the sample answer corresponding to the sample question satisfies the output content constraint, and combine the sample answer that satisfies the output content constraint and its sample question into a question-answer pair.
[0303] In some embodiments, the configuration of the question type template includes at least one of the following:
[0304] In the case of fine-tuning the current generated task instruction model, the question template is configured as at least one of the following: an insulating gas knowledge question and answer template, an experimental condition explanation and reasoning template, an instruction response template, or a multi-round expert dialogue template; wherein, the instruction response template is used to define specific task instructions and compliance response outputs in the field of insulating gases;
[0305] When the current generation task indicates that the model retrieves enhanced generation requirements, the question template is configured as a source-tracing question-answering template. The source-tracing question-answering template includes question-answer pairs and a mapping relationship between the question-answer pairs and the candidate corpus fragments actually referenced by the generated question-answer pairs.
[0306] In the case of the current task instruction model evaluation requirements, the question type template is configured as at least one of the following: judgment question template, single choice question template, multiple choice question template, short answer question template, or open-ended question template, and the question type template is associated with a reference answer template and a scoring point template that match the question type.
[0307] In some embodiments, the target dataset includes a data file and a manifest file; the export module based on the dataset is specifically used for:
[0308] From the multiple question-answer pairs already constructed under the current generation task, determine the subset of question-answer pairs to be exported according to the application requirements of the model indicated by the current generation task;
[0309] The question-answer pair subset is exported as a data file matching the application requirements of the model using an export template; the export template is selected according to the application requirements of the model indicated by the current generation task.
[0310] Generate a manifest file associated with the data file; the manifest file is used to record the metadata of each question-answer pair in the question-answer pair subset.
[0311] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.
[0312] This disclosure also provides an apparatus including a memory, a processor, and a computer program stored in the memory, wherein the processor executes the computer program to implement the steps of the above-described method.
[0313] This disclosure also provides a non-volatile computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described method.
[0314] Figure 7 This is a block diagram illustrating an apparatus 700 for the aforementioned pipeline network construction method according to an exemplary embodiment. For example, apparatus 700 may be provided as a server or terminal device. (Refer to...) Figure 7 The apparatus 700 includes a processing component 722, which further includes one or more processors, and memory resources represented by memory 732 for storing instructions, such as application programs, that can be executed by the processing component 722. The application programs stored in memory 732 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 722 is configured to execute instructions to perform the methods described above.
[0315] The device 700 may also include a power supply component 726 configured to perform power management of the device 700, a wired or wireless network interface 750 configured to connect the device 700 to a network, and an input / output interface 758 (I / O interface). The device 700 can operate on an operating system stored in memory 732, such as Windows Server™, Mac OSX™, Unix™, Linux™, FreeBSD™, or similar.
[0316] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 732 including computer program instructions that can be executed by the processing component 722 of the device 700 to perform the above-described method.
[0317] Computer-readable storage media can be tangible devices capable of holding and storing programs / instructions used by instruction execution devices. Computer-readable storage media can be, for example—but not limited to—electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.
[0318] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.
[0319] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or technical improvements to the embodiments in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.
Claims
1. A method for generating datasets for models in the field of insulating gases, characterized in that, The method includes: For each original technical document related to the field of insulating gases, the original technical document is parsed into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique identifier of the original technical document and the position information of the corpus unit in the original technical document; The corpus content of each corpus unit parsed from the original technical data is semantically segmented into at least one candidate corpus fragment; any candidate corpus fragment represents a structural-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gas; the candidate corpus fragment carries fragment source information; the fragment source information is determined based on the corpus unit to which the corpus content in the candidate corpus fragment belongs and the unit source information it carries; Determine the corpus tags for each candidate corpus segment; the corpus tags include at least one tag node in the insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize the knowledge keywords or technical topics of the insulating gas domain. Based on the label range of the current generation task, candidate corpus fragments whose labels match the label range are selected, and multiple question-answer pairs are constructed based on the generation constraints of the current generation task and the selected candidate corpus fragments; the current generation task is determined based on the application requirements of the model. Based on the multiple question-answer pairs already constructed under the current generation task, construct the target dataset corresponding to the current generation task.
2. The method according to claim 1, characterized in that, The process of dividing the corpus content of each corpus unit parsed from the original technical data into at least one candidate corpus segment includes: When the original technical data is found to contain experimental facts, the experimental data describing the experimental facts in each corpus unit is identified and extracted, and the experimental data from different corpus units that belong to the same experimental fact are merged to generate the first corpus segment corresponding to the same experimental fact; different experimental facts correspond to different first corpus segments. For the remaining corpus content of each corpus unit excluding the experimental corpus content, the corpus is segmented according to a partitioning strategy to generate at least one second corpus fragment; the first corpus fragment and the second corpus fragment constitute each candidate corpus fragment obtained by semantic segmentation of each corpus unit; the partitioning strategy includes: structural segmentation according to the title level of each corpus unit.
3. The method according to claim 1, characterized in that, The question-answer pair is associated with a source fragment; the source fragment includes candidate corpus fragments actually referenced in generating the question-answer pair; the construction of the target dataset corresponding to the current generation task based on multiple question-answer pairs constructed under the current generation task includes: For each question-and-answer pair, data validation is performed; the data validation includes at least: verifying whether the question-and-answer pair can be supported by the source fragment associated with the question-and-answer pair; and / or verifying whether the question-and-answer pair mixes conclusions from different gas systems, different equipment types, or different test conditions; The target dataset is obtained by integrating the question-answer pairs that have passed the data verification.
4. The method according to claim 1, characterized in that, The question-answer pair is configured with metadata; the metadata includes at least attribute fields and application adaptation fields; the values of the fields in the metadata are determined based on the generation constraints; the attribute fields include at least one of the following: sample label, question type, difficulty level, and conversation round; the sample label is determined based on the corpus labels of the candidate corpus fragments actually referenced in generating the question-answer pair; the application adaptation fields include at least one of the following: the type of expert role simulated when generating the question-answer pair, the target audience of the question-answer pair, and the model application requirements adapted to the question-answer pair. After constructing multiple question-answer pairs based on the selected candidate corpus fragments, the method further includes: Based on the metadata of each of the multiple question-answer pairs, the number of question-answer pairs of different categories under the target statistical dimension is determined; the target statistical dimension is composed of at least one field in the metadata. Based on the number of question-answer pairs in different categories under the target statistical dimension, the target categories that need to be supplemented with question-answer pairs are identified, and question-answer pairs that meet the target categories are supplemented in a targeted manner based on the selected candidate corpus fragments.
5. The method according to claim 1 or 4, characterized in that, The generation constraints include at least question type templates and output content constraints; the output content constraints are used to limit the conversational structure of the question-answer pairs and / or the content expression style of the sample answers; the generation constraints based on the current generation task construct multiple question-answer pairs according to the selected candidate corpus fragments, including: The selected candidate corpus fragments, the output content constraints, and the question type template are used to input the question generation model to generate various sample questions that match the question type template. The answerability of the sample question is verified based on the candidate corpus fragments actually referenced in generating the sample question. The answer generation model is jointly inputted with each sample question that passes the answerability check, the selected candidate corpus fragments, and the output content constraints to generate a sample answer for each sample question. Verify whether the sample answer corresponding to the sample question satisfies the output content constraint, and combine the sample answer that satisfies the output content constraint and its sample question into a question-answer pair.
6. The method according to claim 5, characterized in that, The configuration of the question type template includes at least one of the following: In the case of fine-tuning the current generated task instruction model, the question template is configured as at least one of the following: an insulating gas knowledge question and answer template, an experimental condition explanation and reasoning template, an instruction response template, or a multi-round expert dialogue template; wherein, the instruction response template is used to define specific task instructions and compliance response outputs in the field of insulating gases; When the current generation task indicates that the model retrieves enhanced generation requirements, the question template is configured as a source-tracing question-answering template. The source-tracing question-answering template includes question-answer pairs and a mapping relationship between the question-answer pairs and the candidate corpus fragments actually referenced by the generated question-answer pairs. In the case of the current task instruction model evaluation requirements, the question type template is configured as at least one of the following: judgment question template, single choice question template, multiple choice question template, short answer question template, or open-ended question template, and the question type template is associated with a reference answer template and a scoring point template that match the question type.
7. The method according to claim 4, characterized in that, The target dataset includes data files and manifest files; the construction of the target dataset corresponding to the current generation task based on multiple question-answer pairs already constructed under the current generation task includes: From the multiple question-answer pairs already constructed under the current generation task, determine the subset of question-answer pairs to be exported according to the application requirements of the model indicated by the current generation task; The question-answer pair subset is exported as a data file matching the application requirements of the model using an export template; the export template is selected according to the application requirements of the model indicated by the current generation task. Generate a manifest file associated with the data file; the manifest file is used to record the metadata of each question-answer pair in the question-answer pair subset.
8. A dataset generation device for models in the field of insulating gases, characterized in that, The device includes: The data parsing module is configured to parse each original technical document related to the field of insulating gases into at least one corpus unit; the corpus unit represents a content block with independent semantics in the original technical document; the corpus unit carries unit source information; the unit source information includes the unique identifier of the original technical document and the position information of the corpus unit in the original technical document. The corpus segmentation module is configured to semantically segment the corpus content of each corpus unit parsed from the original technical data into at least one candidate corpus fragment; any candidate corpus fragment represents a structural-semantic composite unit carrying a statement of knowledge or a record of experimental facts in the field of insulating gases; the candidate corpus fragment carries fragment source information; the fragment source information is determined based on the corpus unit to which the corpus content in the candidate corpus fragment belongs and the unit source information it carries; The corpus tag determination module is configured to determine the corpus tag for each candidate corpus segment; the corpus tag includes at least one tag node in an insulating gas domain tag tree; the insulating gas domain tag tree includes a tag hierarchy structure set based on the knowledge organization of the insulating gas domain and the application requirements of the model; any tag node is used to characterize a knowledge keyword or technical topic in the insulating gas domain; The question-answer pair construction module is configured to filter out candidate corpus fragments whose labels match the label range of the current generation task, and construct multiple question-answer pairs based on the generation constraints of the current generation task and the filtered candidate corpus fragments; the current generation task is determined based on the application requirements of the model. The dataset export module is configured to construct the target dataset corresponding to the current generation task based on multiple question-answer pairs already constructed under the current generation task.
9. An electronic device comprising a memory, a processor, and a computer program stored in the memory, characterized in that, The processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.
10. A non-volatile computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.
