Data set construction method and apparatus, and electronic device

Abstract

This application relates to the field of data processing technology and provides a dataset construction method, apparatus, and electronic device, including: acquiring a document; constructing a corresponding knowledge graph based on the document; determining at least two types of entity information based on the knowledge graph; acquiring first prompt words corresponding to various entity information types; for each entity information type, using the entity information as background material for a preset large model, guiding the large model to generate corresponding question-and-answer results through the first prompt words corresponding to the entity information; and determining the corresponding dataset based on the question-and-answer results. This method helps to increase the diversity of data dimensions in the dataset.

Description

Technical Field

[0001] This application belongs to the field of data processing technology, and in particular relates to dataset construction methods, apparatus, electronic devices, computer-readable storage media, and computer program products. Background Technology

[0002] Before applying a model, it is usually necessary to process the model based on the dataset, such as training, testing, and validation.

[0003] Existing datasets are typically constructed using methods such as manual annotation and expert writing. However, constructing datasets using these methods can result in limited knowledge coverage and a single dimension in the final database. Summary of the Invention

[0004] This application provides a dataset construction method, apparatus, and electronic device, which can solve the problem that the datasets generated by existing methods have too few dimensions.

[0005] In a first aspect, embodiments of this application provide a dataset construction method, including: Get the document; Construct a corresponding knowledge graph based on the document; At least two types of entity information are determined based on the knowledge graph; Obtain the first prompt word corresponding to each of the aforementioned entity information; For each entity information, the entity information is used as background material for a pre-defined large model, and the large model is guided to generate corresponding question-and-answer results by the first prompt word corresponding to the entity information. The corresponding dataset is determined based on the question and answer results.

[0006] The beneficial effects of the embodiments in this application compared with the prior art are: In this embodiment, a knowledge graph is constructed based on the acquired documents. At least two types of entity information are determined based on this knowledge graph, and first prompt words corresponding to each type of entity information are obtained. For each entity information, the entity information is used as background material for a pre-defined large model. The pre-defined large model is guided to generate corresponding question-and-answer results using the first prompt words corresponding to the entity information. The corresponding dataset is determined based on the question-and-answer results. Since the question-and-answer results use entity information as background material for the large model, and are generated by the large model through the first prompt words corresponding to the entity information, the large model is subject to dual constraints from both entity information and the first prompt words during the generation of question-and-answer results, thereby reducing the probability of the large model exhibiting illusions. Furthermore, since the entity information is determined from the knowledge graph constructed from the documents, it is beneficial to align the question-and-answer results generated by the large model with the entity nodes of the knowledge graph. Furthermore, since at least two types of entity information are identified from the knowledge graph, and the first prompt words are different for different types of entity information, the question-and-answer results generated by the large model based on different entity information and first prompt words will inevitably differ. Therefore, determining the question-and-answer results based on at least two types of entity information and the first prompt words corresponding to these at least two types of entity information is beneficial to enriching the types of the final dataset, that is, increasing the diversity of the data dimensions of the dataset.

[0007] Secondly, embodiments of this application provide a dataset construction apparatus, including: The document retrieval module is used to retrieve documents; The knowledge graph construction module is used to construct a corresponding knowledge graph based on the document. An entity information determination module is used to determine at least two types of entity information based on the knowledge graph. The first prompt word acquisition module is used to acquire the first prompt word corresponding to various entity information; The question-and-answer result generation module is used to, for each entity information, use the entity information as background material of a preset large model, and guide the large model to generate corresponding question-and-answer results through the first prompt word corresponding to the entity information; The dataset determination module is used to determine the corresponding dataset based on the question and answer results.

[0008] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect.

[0009] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0010] Fifthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in the first aspect.

[0011] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below.

[0013] Figure 1 This is a flowchart illustrating a dataset construction method provided in an embodiment of this application; Figure 2 This is a schematic diagram of a process for determining a dataset Z according to an embodiment of this application; Figure 3 This is a schematic diagram of the structure of a dataset construction apparatus provided in one embodiment of this application; Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0014] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0015] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0016] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0017] Furthermore, in the description of this application and the appended claims, the terms "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0018] References to "one embodiment" or "some embodiments" in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0019] Before deploying a large model, it is usually necessary to first build a dataset for a specific domain and then fine-tune the large model based on this dataset to adapt it to the target task.

[0020] When constructing a dataset, if it is constructed through manual annotation, the annotation team may lack the perspectives of other groups due to their limited background, and strict annotation may also filter out marginal samples. On the other hand, if it is constructed through expert writing, the dataset may be severely homogenized because experts usually specialize in a single field and it is difficult to cover multiple fields.

[0021] That is, when a dataset is constructed in the above manner, it may result in a dataset with a single dimension.

[0022] To increase the dimensionality of the dataset, this application provides a dataset construction method. In this method, at least two types of entity information are determined based on a knowledge graph; first prompt words corresponding to these at least two types of entity information are obtained; each entity information is used as background material for a pre-defined large model; the pre-defined large model is guided to generate corresponding question-and-answer results using the first prompt words corresponding to each entity information; and the corresponding dataset is then determined based on these question-and-answer results.

[0023] The dataset construction method provided in the embodiments of this application is described below with reference to the accompanying drawings.

[0024] Figure 1 A flowchart illustrating a dataset construction method provided in an embodiment of this application is shown. This dataset construction method can be applied to electronic devices, and is described in detail below: S11, retrieve the document.

[0025] The document type mentioned above can include text types such as TXT, CSV, JSON, and XML. Of course, the document type can also be other types, such as image types, web page types, etc., which are not limited here.

[0026] The documents mentioned above can be structured or unstructured; no specific restrictions are imposed here.

[0027] The number of documents mentioned above may be greater than or equal to 1. In this embodiment of the application, the documents may be obtained from a local database or through a network when the electronic device communicates with other devices. No limitation is made here.

[0028] S12, Construct the corresponding knowledge graph based on the above documents.

[0029] In this embodiment of the application, the document can be cleaned before constructing the knowledge graph based on the document. This cleanup includes: removing irrelevant noise from the document (such as removing HTML tags, special characters, headers and footers, garbled text, etc.), and standardizing the format of the noise-removed document to ensure the purity of the document content and provide a high-quality data foundation for subsequent processing.

[0030] In this embodiment, the document (or cleaned document) is segmented into multiple text units, and entities and relationships between entities are extracted from each text unit. The segmented text units are input into a preset entity recognition model, which identifies key entities and their attributes (i.e., information describing entity features, properties, or states) within the text units, determines semantic relationships between entities (such as belonging, cause, location, etc.), and outputs structured triple data: <head entity, relation, tail entity>. An index relationship is established between the triples and the corresponding text units. Finally, a corresponding knowledge graph is constructed based on the triple data (i.e., entities and relationships between entities) and the corresponding text units. Entities serve as nodes in the knowledge graph, and the relationship between two entities serves as an edge connecting the two nodes corresponding to those two entities.

[0031] In this embodiment, text segmentation of a document can employ a sliding segmentation strategy with a fixed window size, such as setting a fixed number of characters (e.g., 512 tokens) and an overlap rate (e.g., 10%) for segmentation. Since the sliding segmentation strategy has lower computational complexity and faster processing speed, it is suitable for scenarios with high requirements for construction efficiency and relatively regular document structures. Optionally, text segmentation of questions and answers can also employ a segmentation strategy combining heading levels with a fixed token sequence length. Of course, other segmentation strategies can also be used, and are not limited here.

[0032] When employing a segmentation strategy combining title hierarchy with a fixed token sequence length, the document is first segmented by title hierarchy. If the segmented units exceed the upper limit of the fixed token sequence length, further segmentation continues using that upper limit to ensure that the segmented text units do not exceed the window limit of the large model used to generate question-answering results. Because this segmentation strategy considers both document titles and the window limit of the large model, it can segment a document (i.e., long text) into several logically independent text units, each containing one or more complete semantic paragraphs. In other words, this segmentation strategy can maintain the semantic integrity of the segmented text units as much as possible. The fixed token sequence length can be determined based on the maximum number of tokens that the large model used to generate question-answering results can process in a single inference process.

[0033] After segmenting the text into multiple text units, the metadata of each text unit (such as source document ID, page number, starting position, etc.) is recorded so that knowledge can be traced based on the metadata of the text unit later.

[0034] In this embodiment, after constructing a knowledge graph based on triple data (i.e., entities and relationships between entities) and the corresponding text units, community detection can be performed on the knowledge graph. Specifically, based on the extracted triple data, graph algorithms (such as the Leiden algorithm or the Louvain algorithm) are applied to perform community detection on the knowledge graph, aggregating closely related entities into different communities. For each community, community information is generated from all associated original text units within that community, and the generated community information is stored, such as storing it as an entity attribute or as an entity tag. The aforementioned community information may include one or more of the following: a unique community identifier, a community name, a member list, and a community summary. The community summary summarizes the core themes and key information chains within the community, providing a macro-context for subsequent high-level reasoning.

[0035] S13, determine at least two types of entity information based on the above knowledge graph.

[0036] The aforementioned entity information refers to information related to an entity, such as the entity name, the name of another entity related to the entity, the relationship between the two entities, etc., all of which can be considered as the aforementioned entity information.

[0037] In this embodiment of the application, different types of entity information contain different information types. For example, assuming that one type of entity information includes an entity and the corresponding document information (or text unit), another type of entity information may include an entity pair, the relationship between the entity pair, and the document information of the entity pair.

[0038] That is, in this embodiment of the application, at least two types of entity information are determined based on the above knowledge graph, including: Based on the entities in the knowledge graph and the document information corresponding to the entities (i.e., the text units corresponding to the entities), determine one type of entity information.

[0039] The entities mentioned above can be specified by the user, in which case the electronic device receives the entity input by the user through the input device; alternatively, the entities can be obtained through filtering by a preset large model, in which case the electronic device acquires the entities filtered by the preset large model. For example, the preset large model can filter out isolated nodes in the knowledge graph, and the entity corresponding to the isolated node is used as the entity information of the above entity; or, for example, the preset large model can filter out high-order nodes in the knowledge graph (i.e., nodes with highly connected paths), and the entity corresponding to the high-order node is used as the entity information of the above entity. Once the entity is determined, the text unit corresponding to the entity information can be determined according to the correspondence between the entity and the text unit.

[0040] Alternatively, based on the aforementioned knowledge graph, at least two types of entity information can be identified, including: Based on the entity pairs in the knowledge graph, the relationships between the entity pairs, and the document information of the entity pairs (i.e., the text units corresponding to the entities in the entity pairs), determine one type of entity information.

[0041] In this embodiment, an entity pair comprises two entities, and the relationship between the entity pair is the relationship between the two entities. The document information of the entity pair consists of the text units corresponding to the two entities respectively. Optionally, the entity pair can be specified by the user or obtained through filtering using a preset large model; this will not be elaborated further here. Once an entity pair is determined, the relationship corresponding to the entity pair can be determined based on the data of the triples, and the text units corresponding to the two entities in the entity pair can be determined based on the correspondence between entities and text units.

[0042] Alternatively, based on the aforementioned knowledge graph, at least two types of entity information can be identified, including: After performing community detection on the knowledge graph and generating community information for the detected communities, an entity information is determined based on two target entities in the knowledge graph, the community information of the two target entities, the relationship between the target entities, and the document information (i.e., the text units corresponding to the target entities) of each of the above target entities, wherein the two target entities are not directly connected.

[0043] The two target entities mentioned above correspond to two different text units. These two target entities may correspond to the same community or different communities.

[0044] In this embodiment, the two target entities mentioned above can also be specified by the user or obtained through filtering by a preset large model, which will not be elaborated here. After the target entities are determined, the community information corresponding to each target entity can be determined according to the correspondence between the target entities and the community, the relationship between the target entities can be determined according to the data of the triples, and the text units corresponding to each target entity can be determined according to the correspondence between the target entities and the text units.

[0045] Alternatively, based on the aforementioned knowledge graph, at least two types of entity information can be identified, including: Two target entities are identified in the knowledge graph above, and these two target entities are not directly connected. Determine the path connecting the two target entities mentioned above; Based on the two target entities and the path connecting them, determine the entity information of the knowledge graph.

[0046] In this embodiment, two entities without direct connection can be specified in the knowledge graph. These two entities are designated as target entities, with the first specified target entity serving as the head target entity and subsequent target entities serving as the tail target entities. After determining the head and tail target entities, graph search algorithms (such as breadth-first search (BFS) or depth-first search (DFS)) can be used to find the shortest path or multiple paths connecting the head and tail target entities. All intermediate relationships along the found paths, i.e., the corresponding original text units, are then concatenated to obtain entity information for one type of knowledge graph.

[0047] Optionally, when determining at least two types of entity information based on a knowledge graph, the entity hop count can also be used. For example, when the entity hop count is 1, the entity information involves one entity and another entity (or attribute node) directly connected to that entity; in this case, these entity information pieces correspond to the same text unit. When the entity hop count is 2, the entity information involves the neighbor's neighbor of the entity; in this case, the text units corresponding to these entity information pieces are usually 2, and the large model also needs to understand the relationships between these text units. When the entity hop count is P (P is greater than 2), the entity information corresponds to multiple text units, and the large model needs to perform long-chain reasoning. Determining entity information through specific entity hop counts is beneficial for improving the refinement of evaluation metrics and for establishing continuous evaluation benchmarks.

[0048] In this embodiment, when the determined entity information includes the document information corresponding to the entity, all subsequent question-and-answer results generated based on that entity information undergo fact-checking, thereby ensuring that the question-and-answer results are definite and verifiable, avoiding the illusion problem that may occur when generating data using traditional large models. Simultaneously, this mechanism provides a cleansing evaluation and tracing path; when the preset large model answers incorrectly, the specific reference text can be quickly located for error analysis.

[0049] S14, obtain the first prompt word corresponding to each of the above entity information.

[0050] The aforementioned first prompt word can be determined based on information input by the user. For example, after the user inputs information on the preset prompt word input interface of the large model, the electronic device uses that information as the aforementioned first prompt word.

[0051] Optionally, the aforementioned first prompt word can also be generated based on a preset large model. For example, the user uses the requirements of the question-and-answer construction instruction corresponding to the entity information as input to the preset large model. The preset large model outputs one or more candidate prompt words according to the requirements of the question-and-answer construction instruction for the user to confirm. The electronic device uses the candidate prompt words confirmed by the user as the aforementioned first prompt word.

[0052] In this embodiment of the application, a corresponding first prompt word can be obtained for each type of entity information, that is, the first prompt word is related to the type of entity information.

[0053] Optionally, for entity information that includes both the entity and its corresponding document information, the requirement for the question-and-answer instruction for this type of entity information can be: The large model is required to generate factual question-and-answer pairs around the concept of the entity based on the entity's document information. Assuming the entity in this entity information is entity A, the first prompt word satisfying the requirement of this type of entity information question-and-answer instruction could be: Please generate questions and answers related to the concept of entity A based on the document information of entity A. Of course, the generated answers can include only the standard answers recognized by the large model, or they can include both the standard answers recognized by the large model and incorrect answers not recognized by the large model; this is not limited here.

[0054] Optionally, for entity information that includes entity pairs, the relationships between those entity pairs, and the document information of those entity pairs, the requirement for the question-and-answer instruction for this type of entity information can be: The large model is required to generate a question and answer describing the relationship between entity pairs based on the document information of the entity pairs. Assuming the entity pair in this entity information is entity pair B, the first prompt word satisfying the requirement of the question-and-answer instruction for this type of entity information could be: Please generate a question and answer that deeply examines the relationship between entity pair B based on the document information corresponding to entity pair B. Of course, the generated answer can include only the standard answer recognized by the large model, or it can include both the standard answer recognized by the large model and incorrect answers not recognized by the large model; this is not limited here.

[0055] Optionally, for entity information including two target entities without a direct connection, community information of the two target entities, the relationship between the target entities, and document information of each of the aforementioned target entities, the requirement for question-and-answer instructions for this type of entity information can be: requiring the large model to integrate information and perform logical deduction based on information fragments scattered across different document information to generate reasoning-based questions and answers. Assuming the entities in this entity information are target entity C and target entity D, the first prompt word satisfying the requirement for question-and-answer instructions for this type of entity information could be: understanding the content and relationship between the text units corresponding to target entity C and the text units corresponding to target entity D. Of course, the generated answers can include only the standard answers recognized by the large model, or they can include both the standard answers recognized by the large model and incorrect answers not recognized by the large model; this is not limited here.

[0056] Optionally, for entity information that includes two target entities and the path connecting the two target entities, the requirement for the question-answering instruction of this type of entity information can be: requiring the large model to generate a path connecting the target entities based on the path of the target entities. Assuming that the entities in this entity information are target entity E and target entity F, then the first prompt word that satisfies the requirement of the question-answering instruction of this type of entity information can be: based on the path corresponding to target entity E and target entity F, generate a reasoning question about how to reach target entity F from target entity E.

[0057] S15, for each of the above entity information, using the above entity information as the background material of the preset large model, the above large model is guided to generate the corresponding question and answer results by the above first prompt words corresponding to the above entity information.

[0058] The aforementioned pre-defined large model refers to a high-performance, analytically capable large model. This large model may include a Large Language Model (LLM), which is an ultra-large-scale neural network model based on the Transformer architecture and pre-trained on massive amounts of text.

[0059] The question-and-answer results described above can take the form of question-and-answer pairs, in which case the result includes one question and one answer. Optionally, the results can also take the form of multiple-choice questions, in which case the result includes one question and multiple options, which can include simple or difficult distractors. Simple distractors can be randomly selected from the knowledge graph as incorrect options that are not directly related to the correct entity, while difficult distractors are selected from the knowledge graph as incorrect options that belong to the same community and have the same relationship type as the correct entity. Because the distractors generated using the knowledge graph have extremely high semantic similarity, they can effectively test the ability of large models to distinguish subtle differences. Of course, the question-and-answer results described above can also take other forms, which are not limited here.

[0060] In this embodiment, the user can import entity information into the large model through the background material addition entry (such as the "+" button). The large model parses the entity information and locates the text fragment that can be used as the answer in the document information included in the entity information based on the parsing result and the first prompt word. The located text fragment is converted into a natural language question, and it is ensured that the question and answer are semantically matched and that the answer can be supported by the background material. After that, the electronic device obtains the question and answer output by the large model.

[0061] In this embodiment, each prompt word corresponds to a question-and-answer result. To more clearly describe how to generate questions and answers based on the first prompt word, the following description uses specific examples: Suppose we randomly sample a triple with a clear relationship <head entity, relation, tail entity> in a knowledge graph. The document information entity pair involved in this triple is: "Data Classification and Grading" and "Cybersecurity Law of the People's Republic of China". The relationship between these entity pairs is: The data classification and grading process is based on the relevant requirements of the "Cybersecurity Law of the People's Republic of China". The document information for this entity pair is: Data classification and grading is the process of classifying data according to certain standards and assigning corresponding security levels. It is an important part of information security management in enterprises, organizations, and governments. Article 21, paragraph 4 of the "Cybersecurity Law of the People's Republic of China" explicitly requires: Measures such as data classification, important data backup, and encryption should be taken to prevent data leakage, theft, and tampering. The first prompt is: You are a professional knowledge-based question-and-answer generation expert. Based on the above document information, please generate a question and answer that deeply examines the relationship between the entities "Data Classification and Grading" and "Cybersecurity Law of the People's Republic of China".

[0062] Question: What regulations govern the data classification and grading process? Answer: The data classification and grading process is based on Article 21, Paragraph 4 of the Cybersecurity Law of the People's Republic of China.

[0063] Suppose we select a community in a knowledge graph that contains multiple nodes (e.g., at least two entity pairs) and edges (i.e., the relationships between entity pairs). We obtain the entity set within this community (assumed to be: personal information, data classification and grading, personal information classification and data grading table, and the "Cybersecurity Law of the People's Republic of China"), the relationship set (assumed to be: (personal information, data classification and grading), (data classification and grading, the "Cybersecurity Law of the People's Republic of China"), and (personal information, personal information classification and data grading table)), as well as the aforementioned generated community information (assumed to be named: "Data Classification and Grading and Cloud Service Data Security Specification Community"). Simultaneously, we retrieve all relevant original text units covered by this community (assumed to be: "Data Classification and Grading.md", "Personal Information Classification and Grading.md", and "Cloud Service Specification.md"). The first prompt could be: understanding the content and relationship between the cloud service specification and the data classification and grading documents.

[0064] Q&A: Question: How does data classification and grading guide the formulation of cloud service data security specifications? Answer: Using 'data classification and grading' as the core process / specification as the hub, and based on the requirements of the "Cybersecurity Law of the People's Republic of China," it is specifically refined into 'Cloud Service Data Classification and Grading and Collection, Storage, and Access Specifications.' This community focuses on the definition, association, and compliance management of various data models (such as personal information, sensitive personal information, and data of different levels), forming a multi-layered governance structure from laws and regulations, overall processes to specific implementation specifications and data standards. The core association methods are legal basis, process instantiation, object attribution, and level mapping.

[0065] S16. Determine the corresponding dataset based on the above question and answer results.

[0066] In this embodiment, the electronic device can store question-and-answer results corresponding to entity information of the same type into the same dataset. This allows for subsequent evaluation of different capabilities of the large model based on different datasets. For example, suppose dataset X stores question-and-answer results for entity information containing entities and their corresponding document information; dataset Y stores question-and-answer results for entity information containing entity pairs, relationships between entity pairs, and document information of entity pairs; and dataset Z stores question-and-answer results for entity information containing two unconnected target entities, community information of the two target entities, relationships between target entities, and document information of each target entity. Since dataset X stores factual question-and-answer results, dataset Y stores entity relationship question-and-answer results, and dataset Z stores inference-based question-and-answer results, dataset X can be used to evaluate the large model's memory capacity, dataset Y can be used to evaluate the large model's semantic understanding and association capabilities, and dataset Z can be used to evaluate the large model's cross-document comprehensive analysis and complex reasoning capabilities.

[0067] In this embodiment, a knowledge graph is constructed based on the acquired documents. At least two types of entity information are determined from this knowledge graph, and first prompt words corresponding to each type of entity information are obtained. For each entity information, the entity information serves as background material for a pre-defined large model. The pre-defined large model is guided to generate corresponding question-and-answer results using the first prompt words corresponding to the entity information. The corresponding dataset is then determined based on these question-and-answer results. Since the question-and-answer results use entity information as background material for the large model, and are generated by the large model using the first prompt words corresponding to the entity information, the large model is subject to dual constraints from both entity information and the first prompt words during the question-and-answer result generation process, thereby reducing the probability of the large model exhibiting illusions. Furthermore, since the entity information is determined from the knowledge graph constructed from the documents, it facilitates the alignment of the question-and-answer results generated by the large model with the entity nodes of the knowledge graph. Furthermore, since at least two types of entity information are identified from the knowledge graph, and the first prompt words are different for different types of entity information, the question-and-answer results generated by the large model based on different entity information and first prompt words will inevitably differ. Therefore, determining the question-and-answer results based on at least two types of entity information and the first prompt words corresponding to these at least two types of entity information is beneficial to enriching the types of the final dataset, that is, increasing the diversity of the data dimensions of the dataset.

[0068] In the above description, the electronic device can directly determine the corresponding dataset based on the question-and-answer results generated by the preset large model. In some embodiments, considering that the question-and-answer results may have low accuracy or low completeness, the question-and-answer results generated by the large model can be filtered before determining the dataset. That is, after guiding the preset large model to generate corresponding question-and-answer results through the first prompt words corresponding to the above entity information, the method further includes: Determine the second prompt word, which includes evaluation information based on preset evaluation dimensions; The second prompt word above guides the large model to evaluate the question-and-answer results. Based on the evaluation results, select the question and answer results that meet the preset conditions from the above question and answer results to obtain the target question and answer results; Correspondingly, the dataset determined based on the above question-and-answer results includes: The dataset is determined based on the above target question and answer results.

[0069] In this embodiment, the evaluation dimensions can be set according to actual requirements, such as based on different datasets. For example, for the question-and-answer results corresponding to dataset X, the evaluation dimensions may include: accuracy, completeness, coherence, and entity fact accuracy of the answer. For example, for the question-and-answer results corresponding to dataset Y, the evaluation dimensions may include: accuracy, completeness, coherence, entity fact accuracy, and relation accuracy of the answer. For example, for the question-and-answer results corresponding to dataset Z, the evaluation dimensions may include: accuracy, completeness, coherence, entity fact accuracy, relation accuracy, and reasoning path accuracy of the answer. Since corresponding evaluation dimensions are set according to different datasets, it is beneficial to improve the accuracy of the set evaluation dimensions, thereby improving the accuracy of the question-and-answer results selected based on higher-precision evaluation dimensions.

[0070] In this embodiment of the application, after determining the corresponding evaluation dimension for the current output question and answer result, the corresponding second prompt word is determined according to the evaluation information of the evaluation dimension. For example, assuming that the evaluation dimension includes accuracy, and the evaluation information of the accuracy includes a score of 0 to 100, the determined second prompt word can be: Please evaluate the accuracy of the current output question and answer result based on the evaluation information. At this time, the evaluation result obtained is the specific score corresponding to the accuracy.

[0071] In this embodiment, the preset conditions correspond to the evaluation results. When the evaluation results are represented by scores, the preset conditions may include: a score exceeding m1 (where m1 is a specific score), or scores ranking in the top N1 (where N1 is a natural number greater than 0). When the evaluation results are represented by grades, the preset conditions may include: a grade exceeding m2 (where m2 is a specific score), or grades ranking in the top N2 (where N2 is a natural number greater than 0). Of course, the evaluation results may also be represented in other ways, which are not limited here.

[0072] In this embodiment of the application, since the dataset is determined by filtering the question and answer results, it is beneficial to improve the quality of the dataset.

[0073] In this embodiment of the application, after determining the dataset, the dataset can be evaluated. That is, after determining the dataset based on the question-and-answer results, the process further includes: Perform sample-level evaluation on the above datasets, and / or perform dataset-level evaluation on the above datasets.

[0074] The aforementioned sample set evaluation refers to the quality or performance evaluation of each (or sampled) data sample (such as a single question-and-answer pair), which can be judged on a scale of 0 to 10.

[0075] The aforementioned dataset evaluation refers to a comprehensive evaluation of the statistical characteristics, distribution quality, and overall performance of the entire dataset.

[0076] In this embodiment of the application, when performing sample-level evaluation on the dataset, erroneous samples can be precisely located at the micro level. Conversely, when performing dataset-level evaluation, problems at the overall dataset level (such as bias, leakage, and duplication) can be identified at the macro level.

[0077] Optionally, when evaluating a sample set, the evaluation can be performed from the following dimensions: (1) Consistency of facts: Definition: Whether the answer is objectively true and whether it contains hallucinations.

[0078] Evaluation method: Given the text unit containing the entity, check whether the key entities, values, and dates in the answer are reasonably present in the context or general knowledge base through a large model.

[0079] (2) Logical consistency: Definition: If the dataset contains thought processes, are the reasoning steps rigorous, and are there logical contradictions in the description of the question-and-answer results?

[0080] Evaluation method: The large model evaluates whether there are any inconsistencies in the description of the question-and-answer results.

[0081] (3) Instruction compliance: Definition: Whether the output format, word limit, and style requirements conform to the instructions.

[0082] Evaluation method: Regular expression parsing combined with semantic judgment of large model is used to check whether the constraints are met.

[0083] (4) Complexity calibration: Definition: Whether the actual difficulty of a sample is consistent with its labeled difficulty level (e.g., easy / medium / difficult).

[0084] Evaluation method: The cognitive load required to solve the problem is analyzed using a large model and compared with the original label.

[0085] Optionally, when evaluating a dataset, the evaluation can be performed from the following dimensions: (1) Coverage and diversity: Implementation: Semantic clustering is performed on the "question" portion of the dataset. The large model is then evaluated to generate "topic labels" for each cluster.

[0086] Evaluation: Calculate the inter-cluster distance and intra-cluster compactness of the clusters. High intra-cluster similarity indicates a large amount of "template-based" data, suggesting low quality.

[0087] (2) Safety assessment: Implementation: Evaluate the large model as a filter to scan the dataset for bias, toxic content, or personal privacy information (PII).

[0088] Output: The proportion and type distribution of risk samples. Typical characteristics of high security include: a low proportion of total risk samples (e.g., less than 0.05%), and no PII-type risks; only a very small number of Bias-type and Toxic Content-type risks, with no harmful content; risk samples exhibiting no clustering, randomly distributed throughout the dataset, and no batches of homogeneous risks.

[0089] (3) Data redundancy: Implementation: Combining embedding similarity and semantic judgment from a large model, highly similar "synonymous repetition" samples are identified. Embedding refers to a method of converting question-and-answer results into a low-dimensional dense vector representation, making semantically similar content closer together in the vector space.

[0090] Assessment: Calculate the percentage of duplicate samples. When the percentage of duplicate samples exceeds a preset threshold, it indicates that the dataset has high data redundancy.

[0091] To more clearly describe how to determine the dataset, the following description will take determining the dataset Z as an example.

[0092] refer to Figure 2 , Figure 2 A schematic diagram of a process for determining a dataset Z according to an embodiment of this application is shown.

[0093] S21, retrieve the document.

[0094] S22, Construct the corresponding knowledge graph based on the above documents.

[0095] The process of constructing the knowledge graph is detailed in the description above and will not be repeated here.

[0096] S23, determine entity information based on the two target entities in the knowledge graph, the community information of the two target entities, the relationship between the target entities, and the document information of each target entity, wherein the two target entities are not directly connected.

[0097] As described above, dataset Z stores the question-and-answer results corresponding to entity information, which includes two target entities that are not directly connected, the community information of the two target entities, the relationship between the target entities, and the document information of each target entity. Therefore, the entity information determined here is based on the two target entities that are not directly connected, the community information of the two target entities, the relationship between the target entities, and the document information of each target entity.

[0098] S24, obtain the first prompt word corresponding to the above entity information.

[0099] S25, using the aforementioned entity information as background material for a pre-defined large model, the large model is guided to generate corresponding question-and-answer results by using the aforementioned first prompt words corresponding to the aforementioned entity information.

[0100] S26, determine the second prompt word that includes evaluation information of the preset evaluation dimensions.

[0101] S27, using the second prompt word mentioned above, guide the large model to evaluate the question-and-answer results.

[0102] S28. Based on the evaluation results, select the question and answer results that meet the preset conditions from the above question and answer results to obtain the target question and answer results.

[0103] S29. Determine the dataset Z based on the above target question and answer results.

[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0105] Corresponding to the dataset construction method described in the above embodiments, Figure 3 The diagram shows a structural block diagram of a dataset construction apparatus provided in an embodiment of this application. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0106] Reference Figure 3 The dataset construction device 3 is applied to an electronic device and includes: a document acquisition module 31, a knowledge graph construction module 32, an entity information determination module 33, a first prompt word acquisition module 34, a question-and-answer result generation module 35, and a dataset determination module 36. Among them: Document retrieval module 31 is used to retrieve documents.

[0107] The knowledge graph construction module 32 is used to construct the corresponding knowledge graph based on the above documents.

[0108] In this embodiment of the application, the document can be cleaned before constructing the knowledge graph. This cleaning includes removing irrelevant noise from the document and standardizing the format of the noise-removed document to ensure the purity of the document content and provide a high-quality data foundation for subsequent processing.

[0109] In this embodiment of the application, the text segmentation of the document can be performed using a sliding segmentation strategy with a fixed window size, or a segmentation strategy combining the title level with a fixed token sequence length.

[0110] The entity information determination module 33 is used to determine at least two types of entity information based on the knowledge graph mentioned above.

[0111] The first prompt word acquisition module 34 is used to acquire the first prompt word corresponding to the various entity information mentioned above.

[0112] The question-and-answer result generation module 35 is used to, for each of the above entity information, use the above entity information as the background material of the preset large model, and guide the above large model to generate the corresponding question-and-answer result through the above first prompt words corresponding to the above entity information.

[0113] The dataset determination module 36 is used to determine the corresponding dataset based on the above question and answer results.

[0114] In this embodiment, a knowledge graph is constructed based on the acquired documents. At least two types of entity information are determined based on this knowledge graph, and first prompt words corresponding to each type of entity information are obtained. For each entity information, the entity information serves as background material for a pre-defined large model. The pre-defined large model is guided to generate corresponding question-and-answer results using the first prompt words corresponding to the entity information. The corresponding dataset is then determined based on the question-and-answer results. Since the question-and-answer results use entity information as background material for the large model, and are generated by the large model through the first prompt words corresponding to the entity information, the large model is subject to dual constraints from both entity information and the first prompt words during the generation of question-and-answer results, thereby reducing the probability of the large model exhibiting illusions. Furthermore, since the entity information is determined from the knowledge graph constructed from the documents, it facilitates the alignment of the question-and-answer results generated by the large model with the entity nodes of the knowledge graph. Furthermore, since at least two types of entity information are identified from the knowledge graph, and the first prompt words are different for different types of entity information, the question-and-answer results generated by the large model based on different entity information and first prompt words will inevitably differ. Therefore, determining the question-and-answer results based on at least two types of entity information and the first prompt words corresponding to these at least two types of entity information is beneficial to enriching the types of the final dataset, that is, increasing the diversity of the data dimensions of the dataset.

[0115] Optionally, the entity information determination module 33 is specifically used for: Based on the entities in the knowledge graph and the document information corresponding to those entities, determine one type of entity information.

[0116] Optionally, the entity information determination module 33 is specifically used for: Based on the entity pairs in the knowledge graph, the relationships between the entity pairs, and the document information of the entity pairs, determine one type of entity information.

[0117] Optionally, the dataset construction apparatus 3 provided in this application embodiment further includes: The community detection module is used to perform community detection on the knowledge graph after the knowledge graph is constructed based on the above documents. The community information generation module is used to generate community information for detected communities; Correspondingly, the entity information determination module 33 mentioned above is specifically used for: Based on the two target entities in the knowledge graph, the community information of the two target entities, the relationship between the target entities, and the document information of each target entity, one type of entity information is determined, wherein the two target entities are not directly connected.

[0118] Optionally, the aforementioned entity information determination module 33 includes: The target entity determination unit is used to determine two target entities in the above knowledge graph, which are not directly connected. The target entity connectivity path determination unit is used to determine the path connecting the two target entities mentioned above; The entity information determination unit is used to determine entity information of the knowledge graph based on the two target entities and the path connecting the two target entities.

[0119] Optionally, the dataset construction apparatus 3 provided in this application embodiment further includes: The second prompt word determination module is used to determine the second prompt word, which includes evaluation information of the preset evaluation dimension, after the preset large model is guided to generate the corresponding question and answer results by the first prompt word corresponding to the above entity information. The question-and-answer result evaluation module is used to guide the large model to evaluate the question-and-answer results based on the second prompt word mentioned above. The target question and answer result filtering module is used to filter out the question and answer results that meet the preset conditions from the above question and answer results based on the evaluation results, and obtain the target question and answer results; Correspondingly, the aforementioned dataset determination module 36 is specifically used for: The dataset is determined based on the above target question and answer results.

[0120] Optionally, the dataset construction apparatus 3 provided in this application embodiment further includes: The dataset evaluation module is used to perform sample-level evaluation and / or dataset-level evaluation on the dataset after the dataset is determined based on the above question and answer results.

[0121] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0122] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 4 As shown, the electronic device 4 of this embodiment includes: at least one processor 40 ( Figure 4 The diagram shows only one processor, memory 41, and computer program 42 stored in the memory 41 and executable on at least one processor 40. When the processor 40 executes the computer program 42, it implements the steps in any of the above method embodiments.

[0123] The aforementioned electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other computing device. This electronic device may include, but is not limited to, a processor 40 and a memory 41. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or combine certain components, or different components. For example, it may also include input / output devices, network access devices, etc.

[0124] The processor 40 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0125] In some embodiments, the aforementioned memory 41 may be an internal storage unit of the electronic device 4, such as a hard disk or memory of the electronic device 4. In other embodiments, the aforementioned memory 41 may be an external storage device of the electronic device 4, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 4. Furthermore, the aforementioned memory 41 may include both internal storage units and external storage devices of the electronic device 4. The aforementioned memory 41 is used to store operating systems, applications, bootloaders, data, and other programs, such as the program code of the aforementioned computer programs. The aforementioned memory 41 may also be used to temporarily store data that has been output or will be output.

[0126] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0127] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0128] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can implement the steps in the above-described method embodiments.

[0129] This application provides a computer program product that, when run on an electronic device, enables the electronic device to implement the steps described in the various method embodiments above.

[0130] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographic device / electronic device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0131] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0132] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0133] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0134] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

Claims

1. A method for constructing a dataset, characterized in that, include: Get the document; Construct a corresponding knowledge graph based on the document; At least two types of entity information are determined based on the knowledge graph; Obtain the first prompt word corresponding to each of the aforementioned entity information; For each entity information, the entity information is used as background material for a pre-defined large model, and the large model is guided to generate corresponding question-and-answer results by the first prompt word corresponding to the entity information. The corresponding dataset is determined based on the question and answer results.

2. The dataset construction method as described in claim 1, characterized in that, The determination of at least two types of entity information based on the knowledge graph includes: The entity information is determined based on the entities in the knowledge graph and the document information corresponding to the entities.

3. The dataset construction method as described in claim 1, characterized in that, The determination of at least two types of entity information based on the knowledge graph includes: The entity information is determined based on the entity pairs in the knowledge graph, the relationships between the entity pairs, and the document information of the entity pairs.

4. The dataset construction method as described in claim 1, characterized in that, After constructing the corresponding knowledge graph based on the document, the method further includes: Community detection is performed on the knowledge graph. Generate community information for the detected communities; The determination of at least two types of entity information based on the knowledge graph includes: An entity information is determined based on two target entities in the knowledge graph, the community information of the two target entities, the relationship between the target entities, and the document information of each target entity, wherein the two target entities are not directly connected.

5. The dataset construction method as described in claim 1, characterized in that, The determination of at least two types of entity information based on the knowledge graph includes: Two target entities are identified in the knowledge graph, and the two target entities are not directly connected. Determine the path connecting the two target entities; Entity information of the knowledge graph is determined based on the two target entities and the path connecting the two target entities.

6. The dataset construction method according to any one of claims 1 to 5, characterized in that, After guiding the preset large model to generate corresponding question-and-answer results using the first prompt word corresponding to the entity information, the method further includes: Determine the second prompt word, which includes evaluation information based on preset evaluation dimensions; The second prompt word guides the large model to evaluate the question-and-answer results; Based on the evaluation results, select the question and answer results that meet the preset conditions from the question and answer results to obtain the target question and answer results; The step of determining the dataset based on the question-and-answer results includes: The dataset is determined based on the target question-and-answer results.

7. The dataset construction method according to any one of claims 1 to 5, characterized in that, After determining the dataset based on the question-and-answer results, the method further includes: Perform sample-level evaluation on the dataset, and / or perform dataset-level evaluation on the dataset.

8. A dataset construction apparatus, characterized in that, include: The document retrieval module is used to retrieve documents; The knowledge graph construction module is used to construct a corresponding knowledge graph based on the document. An entity information determination module is used to determine at least two types of entity information based on the knowledge graph. The first prompt word acquisition module is used to acquire the first prompt word corresponding to various entity information; The question-and-answer result generation module is used to, for each entity information, use the entity information as background material of a preset large model, and guide the large model to generate corresponding question-and-answer results through the first prompt word corresponding to the entity information; The dataset determination module is used to determine the corresponding dataset based on the question and answer results.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program, which, when executed, causes the electronic device to perform the method according to any one of claims 1 to 7.

Images