Corpus generation method, device, equipment and storage medium

Abstract

The present application discloses a corpus generation method, device, equipment and storage medium, which relates to the field of knowledge graph technology, including: obtaining original table data, wherein the original table data is data in a structured library table format; mapping the original table data according to a preset graph representation structure to obtain the mapped original table data; format conversion of the mapped original table data to obtain key-value pair data corresponding to the original table data; receiving a first instruction input from an external source, inputting the first instruction and the key-value pair data into a preset large model, and obtaining a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph of a graph representation structure based on an actual conversion case in the first instruction; and constructing a corpus based on the knowledge graphs corresponding to multiple original table data. The present application realizes the automatic conversion from structured library table data to a knowledge graph.

Description

Technical Field

[0001] The present application relates to the field of knowledge graph technology, and in particular to a corpus generation method, device, equipment and storage medium. Background Art

[0002] In recent years, knowledge graphs, as an important form of knowledge construction and representation, have played a key role in the fields of artificial intelligence and data science.

[0003] Knowledge graphs can organize entities and their relationships in the form of graphs, allowing machines to understand and reason about this information, and thus support various advanced applications such as intelligent search, recommendation systems, and personal assistants. However, most of the original data is still stored in structured tables, which are usually organized in the form of two-dimensional tables, with each row representing a record and each column representing an attribute. However, knowledge graphs use a graph representation format, that is, a graph structure consisting of nodes and edges. Therefore, there are significant data format differences between structured tables and knowledge graphs, making it difficult to directly build knowledge graphs based on structured tables.

[0004] Therefore, how to achieve effective conversion of structured library table data into knowledge graphs is a problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of this application is to provide a corpus generation method, device, equipment and storage medium, aiming to solve the technical problem of how to effectively convert structured library table data into knowledge graphs.

[0006] To achieve the above purpose, the present application proposes a corpus generation method, which comprises:

[0007] Acquire original table data, wherein the original table data is data in a structured library table format;

[0008] Performing mapping processing on the original table data according to a preset graph representation structure to obtain mapped original table data;

[0009] Performing format conversion on the mapped original table data to obtain key-value pair data corresponding to the original table data;

[0010] Receive a first instruction input from an external source, input the first instruction and the key-value pair data into a preset large model, and obtain a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction;

[0011] A corpus is constructed based on the knowledge graphs corresponding to the multiple original table data.

[0012] In one embodiment, the graph representation structure includes entities, attributes, and relationships between entities, and the step of mapping the original table data according to the preset graph representation structure to obtain the mapped original table data includes:

[0013] Determine the entities, attributes and relationships between entities in the original table data that match the graph representation structure, and map the original table data according to the matching entities, attributes and relationships between entities to obtain mapped original table data.

[0014] In one embodiment, the step of determining entities, attributes, and relationships between entities in the original table data that match the graph representation structure includes:

[0015] Determine the entity types corresponding to different types of original table data;

[0016] Determine each row of data in each of the original table data as an entity of the graph representation structure;

[0017] Determine the header information of the original table data corresponding to each entity type, determine the attribute name of each entity according to the header information, and extract the attribute value corresponding to each attribute name in each entity from each original table data, and use the attribute name and attribute value corresponding to each row of data in the original table data as the attribute of the corresponding entity;

[0018] The relationship types between different types of original table data are analyzed, and the relationship between entities is determined according to the relationship types.

[0019] In one embodiment, the first instruction includes at least a role that executes the instruction, the work content of the role, examples corresponding to the work content, and a detailed description of each part of the knowledge graph, wherein the work content is to convert key-value pair data into a knowledge graph with a graph representation structure, and the examples corresponding to the work content are actual conversion cases for executing the work content.

[0020] In one embodiment, the step of converting the format of the mapped original table data to obtain key-value pair data corresponding to the original table data includes:

[0021] Traverse all entities in the mapped original table data and obtain all attributes corresponding to each of the entities;

[0022] Using the attribute name corresponding to each of the attributes as a key and each of the attributes as a value, creating a plurality of key-value pairs corresponding to each of the entities;

[0023] Key-value pair data including all key-value pairs in each of the entities is created according to the multiple key-value pairs corresponding to each of the entities.

[0024] In one embodiment, after the step of mapping the original table data according to the preset graph representation structure to obtain the mapped original table data, the method further includes:

[0025] According to a preset concatenation rule, concatenate each entity of the mapped original table data and the attribute corresponding to the entity into a first string;

[0026] According to the relationship between the entities in the mapped original table data, the entities are concatenated into a second string;

[0027] A knowledge graph is generated according to the first character string and the second character string.

[0028] In one embodiment, after the step of obtaining the original table data, the method further includes:

[0029] Obtain a fine-tuning dataset, wherein the fine-tuning dataset includes a plurality of historical knowledge graphs and a natural language description of original table data corresponding to each of the historical knowledge graphs;

[0030] Determine a pre-trained model, and fine-tune the pre-trained model using the fine-tuning data set to obtain a fine-tuned generative model, wherein the generative model is used to parse the second instruction input externally to generate a knowledge graph corresponding to the natural language description of the original table data;

[0031] Receive a second instruction input from the outside, input the second instruction into the generation model, and obtain the knowledge graph output by the generation model, wherein the second instruction includes a natural language description corresponding to the original table data.

[0032] In addition, to achieve the above-mentioned purpose, the present application also proposes a corpus generation device, the corpus generation device comprising: an acquisition module, used to acquire original table data, wherein the original table data is data in a structured library table format;

[0033] A mapping module, used to perform mapping processing on the original table data according to a preset graph representation structure to obtain mapped original table data;

[0034] A conversion module, used to convert the format of the mapped original table data to obtain key-value pair data corresponding to the original table data;

[0035] an execution module, configured to receive a first instruction input from an external source, input the first instruction and the key-value pair data into a preset large model, and obtain a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction;

[0036] A generation module is used to construct a corpus based on the knowledge graphs corresponding to the multiple original table data.

[0037] In addition, to achieve the above-mentioned purpose, the present application also proposes a corpus generation device, which includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the corpus generation method described above.

[0038] In addition, to achieve the above-mentioned purpose, the present application also proposes a storage medium, which is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the corpus generation method described above are implemented.

[0039] In addition, to achieve the above-mentioned purpose, the present application also provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the steps of the corpus generation method described above are implemented.

[0040] The present application obtains original table data, wherein the original table data is data in a structured library table format; maps the original table data according to a preset graph representation structure to obtain mapped original table data; performs format conversion on the mapped original table data to obtain key-value pair data corresponding to the original table data; receives a first instruction input from the outside, inputs the first instruction and the key-value pair data into a preset large model, and obtains a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph of a graph representation structure based on an actual conversion case in the first instruction; constructs a corpus based on the knowledge graphs corresponding to multiple original table data, and is capable of converting structured library tables into a standardized key-value pair format, and then customizes the construction rules of the knowledge graph through externally input instructions, thereby generating the required knowledge graph through the large model, realizing automatic conversion from structured library table data to knowledge graphs, and constructing a final corpus, which can effectively improve the automation level of data processing and knowledge graph construction. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0042] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0043] Figure 1A flowchart of the first embodiment of the corpus generation method of the present application is provided;

[0044] Figure 2 A brief flowchart of the corpus generation method provided in Example 1 of the present application;

[0045] Figure 3 A schematic diagram of the process flow provided for the second embodiment of the corpus generation method of the present application;

[0046] Figure 4 This is a schematic diagram of the module structure of the corpus generation device in the embodiment of the present application;

[0047] Figure 5 Schematic diagram of the device structure of the hardware operating environment involved in the corpus generation method in the embodiment of the present application.

[0048] The purpose, features and advantages of this application will be further described in conjunction with the embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0049] It should be understood that the specific embodiments described herein are only used to explain the technical solutions of the present application and are not used to limit the present application.

[0050] In order to better understand the technical solution of the present application, a detailed description will be given below in conjunction with the accompanying drawings and specific implementation methods.

[0051] Based on this, the present application embodiment provides a corpus generation method, referring to Figure 1 , Figure 1 This is a flowchart of the first embodiment of the corpus generation method of the present application.

[0052] In this embodiment, the corpus generation method includes steps S10 to S50:

[0053] Step S10, obtaining original table data, wherein the original table data is data in a structured library table format;

[0054] It should be noted that raw table data refers to data directly obtained from a database or other storage system, usually in a structured form, such as a table in a relational database. This type of data usually has a fixed column and row format, where each column represents an attribute or field, and each row represents a record or instance.

[0055] It is understandable that the original table data is extracted from a database or other data source, and this data will serve as a basis for subsequent steps, ensuring that the subsequent steps can receive correct and complete original table data.

[0056] Step S20, mapping the original table data according to the preset graph representation structure to obtain the mapped original table data;

[0057] It should be noted that the graph representation structure is a data model that uses nodes (vertices) and edges (relationships connecting nodes) to represent entities and their relationships with each other. In this structure, nodes represent entities (such as people, places, events, etc.), and edges represent relationships between entities (such as friendships, affiliations, etc.). Nodes and edges can carry additional information, such as labels or attributes.

[0058] As an example, suppose you get a Customers table and an Orders table, where the Customers table contains fields such as CustomerID, FirstName, LastName, and Email, and the Orders table contains fields such as OrderID, CustomerID, ProductID, Quantity, and OrderDate. Then customers and orders can be considered as two different entities. Each customer record is mapped to a customer node, and each order record is mapped to an order node. Create an edge of the "Customer Order" type to connect the customer node and the order node. The customer node can carry attributes such as FirstName, LastName, and Email, and the order node can carry attributes such as OrderDate and Quantity.

[0059] It can be understood that by converting the original table data into a graph representation structure, a transition from simple table data to a more complex and flexible data representation is achieved, which helps to improve the comprehensibility and usability of the data and also provides a solid foundation for subsequent data analysis and processing. Through step S20, data can be more easily understood, thereby supporting more efficient data-driven decision-making.

[0060] In a feasible implementation, the graph representation structure includes entities, attributes, and relationships between entities, and step S20 may include step S21:

[0061] Step S21, determining entities, attributes and relationships between entities in the original table data that match the graph representation structure, mapping the original table data according to the matching entities, attributes and relationships between entities to obtain mapped original table data.

[0062] It should be noted that entities are the basic components of knowledge graphs, representing objects in the real world, such as people, places, events, etc. Attributes describe the characteristics or status of entities, such as a person’s age, the coordinates of a place, etc. Relationships represent the connection between entities, such as “A is B’s father”.

[0063] Optionally, for the mapping process, you first need to clarify which columns or cells of data can be regarded as entities, classify these entities, such as distinguishing between names of people, places, dates, etc., then identify which columns contain attribute information describing the entities, associate these attribute values ​​with the corresponding entities, and finally determine which cells or columns provide relationship information between entities, and define the specific types of these relationships, such as "works at", "located in", etc.

[0064] In a feasible implementation, step S21 may include steps A10 to A40:

[0065] Step A10, determining entity types corresponding to different types of original table data;

[0066] It should be noted that the entity type refers to the category that represents a type of object in the graph representation structure, such as "customer", "order", "product", etc.

[0067] Optionally, analyze the content and structure of the original table data, understand the meaning of each column and the meaning of each row of data, determine which columns represent entities based on the data content, identify which entity types can be extracted from the original table data, define a name or label for each entity type, determine which attributes each entity type contains (i.e., columns in the original table data), and classify the entities in the original table data into corresponding entity types.

[0068] As an example, there is an original table data, which contains customer information, order information and product information. The original table data contains the following: Customer information table: Entity type: "Customer"; Attributes: CustomerID, FirstName, LastName, Email, PhoneNumber. Order information table: Entity type: "Order"; Attributes: OrderID, CustomerID, ProductID, Quantity, OrderDate. Product information table: Entity type: "Product"; Attributes: ProductID, ProductName, Price, Category.

[0069] Step A20, determining each row of data in each original table data as an entity of the graph representation structure;

[0070] It should be noted that in the graph representation structure, an entity refers to a node that represents an object or concept in the real world. For example, an entity can be a person, a place, an event, etc. A unique identifier is a unique string or number that uniquely identifies each entity in the graph representation structure. For example, a unique identifier for an entity can be its ID number.

[0071] Optionally, analyze each row of the original table data to determine which rows represent an entity. Typically, each row of the original table data represents an instance of an entity. Create an entity for each row of data and determine the type of the entity, that is, which entity type the entity belongs to (for example, "customer", "order", "product", etc.). Assign a unique identifier to each entity. Typically, the original table data already contains a field that can be used as a unique identifier, such as "CustomerID", "OrderID", "ProductID", etc. Ensure that the identifiers of all entities are unique. If the original table data does not contain a field that can be directly used as a unique identifier, you need to create a new identifier.

[0072] As an example, there is an original table data containing customer information, as shown in Table 1:

[0073] Table 1

[0074]

[0075] Create an entity for each row of data, that is, create an entity for each customer, and determine the entity type as "Customer". Assign a unique identifier to each entity, here "CustomerID" is used as the unique identifier, entity 1: CustomerID: 1001, entity 2: CustomerID: 1002.

[0076] It can be understood that step A20 provides a basis for subsequent knowledge graph construction by determining each row of data in the original table data as an entity in the graph representation structure and assigning unique identifiers to these entities, thereby ensuring the uniqueness of each entity in the knowledge graph.

[0077] Step A30, determining the header information of the original table data corresponding to each entity type, determining the attribute name of each entity according to the header information, and extracting the attribute value corresponding to each attribute name in each entity from each original table data, and taking the attribute name and attribute value corresponding to each row of data in the original table data as the attribute of the corresponding entity;

[0078] It should be noted that the header information refers to the first row in the original table data, which usually contains the titles or names of each column. These titles or names represent the attribute names of the entity. The attribute names of the entity refer to the names of the characteristics or attributes of the entity, and these names are derived from the header information in the original table data. The attribute value refers to the specific content of an attribute of the entity, and these contents are derived from the values ​​of the corresponding columns in the original table data.

[0079] Optionally, analyze the header information of the original table data, identify the name of each column, and determine which columns belong to the attributes of a specific entity type. Determine the attribute name of the entity based on the header information. For example, if there are column names such as "CustomerID", "FirstName", "LastName" in the header, these column names can be used as the attribute names of the "Customer" entity. Extract the attributes corresponding to the entity attribute names from the original table data. For example, for the "Customer" entity, extract the values ​​corresponding to the columns such as "CustomerID", "FirstName", "LastName" from the table. Associate the extracted attributes with the corresponding entities, ensuring that the attributes of each entity match its entity type and unique identifier.

[0080] As an example, there is a customer information table as shown in Table 2:

[0081] Table 2

[0082]

[0083] The header information includes "CustomerID", "FirstName", "LastName", "Email", and the attribute names of the "Customer" entity are determined as: "CustomerID", "FirstName", "LastName", "Email". The extracted attributes are: Entity 1: CustomerID: 1001, FirstName: John, LastName: Doe, Email: john.doe@example.com; Entity 2: CustomerID: 1002, FirstName: Jane, LastName: Smith, Email: jane.smith@example.com.

[0084] It can be understood that by determining the header information in the original table data and determining the attribute name and attribute value of the entity based on this information, a data foundation is provided for the subsequent knowledge graph construction, ensuring that the entities in the knowledge graph can accurately represent the information in the original data.

[0085] Step A40, analyzing the relationship types between different types of original table data, and determining the relationship between entities according to the relationship types.

[0086] It should be noted that relationship types refer to different types of connections that may exist between entities, and identify the relationship types implicit in the original table data, such as "belongs to", "purchase", "produce", etc. Establishing relationships between entities means applying relationship types to specific entities.

[0087] As an example, there are two original table data, one is the customer information table, and the other is the order details table: 1. Customer information table: Entity type: Customer; Attributes: CustomerID, FirstName, LastName, Email, Phone. 2. Order details table: Entity type: Order; Attributes: OrderID, CustomerID, ProductID, Quantity, OrderDate. By analyzing the above original table data, we can see that customers can place orders, and orders contain customer information and product information. Since the CustomerID field appears in both tables and can serve as a link between the two entities, it can be identified that there is an "order" relationship between Customer and Order, so there is an "order" relationship between the Customer entity and the Order entity. For each Customer entity, find the corresponding Order entity based on CustomerID and establish an "order" relationship, for example: Entity 1: Customer (CustomerID: 1001); Entity 2: Order (OrderID: 2001, CustomerID: 1001); Relationship: Customer (1001) -> ordered -> Order (2001).

[0088] It can be understood that by analyzing the relationship types between different types of original table data and determining the relationship between entities based on these relationship types, key information is provided for building the knowledge graph, ensuring that the entities in the knowledge graph can correctly represent the connections between entities.

[0089] In this embodiment, the entity types corresponding to different types of original table data are determined; each row of data in each original table data is determined as an entity of the graph representation structure, and the unique identifier of each entity is determined; the header information of the original table data corresponding to each entity type is determined, the attribute name of each entity is determined based on the header information, and the attribute value corresponding to each attribute name in each entity is extracted from each original table data, and the attribute name and attribute value corresponding to each row of data in the original table data are used as the attribute of the corresponding entity; the relationship type between different types of original table data is analyzed, and the relationship between each entity is determined based on the relationship type. This embodiment provides a reasonable framework for converting original table data into a knowledge graph.

[0090] Step S30, converting the format of the mapped original table data to obtain key-value pair data corresponding to the original table data;

[0091] It should be noted that key-value data is a data structure in which data is stored in the form of key and value. The key is usually a string that uniquely identifies the data item, while the value can be any type of data, such as a string, number, Boolean value, etc. Each entity in the graph representation structure can be converted into a key-value pair. Generally speaking, the unique identifier of the entity is used as the key, and other attributes of the entity are used as the value. At the same time, the relationship between entities also needs to be represented as a key-value pair.

[0092] It can be understood that by converting the mapped graph representation structure into key-value pair data, further simplification and standardization of the data is achieved, which helps to improve the efficiency and flexibility of data processing.

[0093] In a feasible implementation, step S30 may include steps S31 to S33:

[0094] Step S31, traverse all entities in the mapped original table data, and obtain all attributes corresponding to each entity;

[0095] It should be noted that the mapped raw table data refers to the data that has completed the mapping process from the raw table data to the graph representation structure. This means that each entity has been defined and the attributes of each entity have also been determined. Entities refer to nodes in the graph representation structure that represent objects or concepts in the real world, such as people, places, events, etc. Attributes refer to the names and values ​​of the characteristics or attributes of an entity, such as a person's name, age, etc.

[0096] Step S32, using the attribute name corresponding to each attribute as a key and each attribute as a value, to create multiple key-value pairs corresponding to each entity;

[0097] As an example, there is the following mapped original table data, which contains two entities: Customer and Order. Customer Information Table: Entity Type: Customer; Attributes: CustomerID, FirstName, LastName, Email. Order Details Table: Entity Type: Order; Attributes: OrderID, CustomerID, ProductID, Quantity, OrderDate. Traverse all entities in the Customer Information Table, and for each customer entity, traverse all its attributes CustomerID, FirstName, LastName, Email. Traverse all entities in the Order Details Table. For each order entity, traverse all its attributes OrderID, CustomerID, ProductID, Quantity, OrderDate. The key-value pairs obtained are as follows: For Customer Entity 1: {"CustomerID":"1001"}{"FirstName":"John"}{"LastName":"Doe"}{"Email":"john.doe@example.com"}. For Order entity 1: {"OrderID":"2001"}{"CustomerID":"1001"}{"ProductID":"3001"}{"Quantity":"2"}{"OrderDate":"2023-01-15"}.

[0098] Step S33: creating key-value pair data including all key-value pairs in each entity according to the multiple key-value pairs corresponding to each entity.

[0099] Based on the key-value pairs obtained in step S32, the key-value pair data of each entity is obtained as follows: For customer entity 1: {"CustomerID":"1001","FirstName":"John","LastName":"Doe","Email":"john.doe@example.com"}. For order entity 1: {"OrderID":"2001","CustomerID":"1001","ProductID":"3001","Quantity":"2","OrderDate":"2023-01-15"}.

[0100] It can be understood that this implementation can convert the original table data into key-value pair data, which can improve the flexibility and efficiency of data processing.

[0101] Step S40, receiving a first instruction input from the outside, inputting the first instruction and the key-value pair data into a preset large model, and obtaining a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction;

[0102] It should be noted that the first instruction is an external input provided by the user or the system to guide the model on how to convert the key-value pair data into a knowledge graph with a graph representation structure. The first instruction usually contains conversion rules, example cases or other guidance information to ensure that the model can correctly understand and process the key-value pair data. The preset large model refers to a model that has been trained or fine-tuned, which can receive key-value pair data and the first instruction and generate a knowledge graph with a graph representation structure.

[0103] It can be understood that by receiving the first instruction and key-value pair data, using the preset large model to convert the data into a knowledge graph with a graph representation structure, and using the guidance information of external input, it is ensured that the model can accurately understand the data structure and generate a knowledge graph that meets the requirements, which helps to improve the interpretability and usability of the data and provides an important data foundation for subsequent corpus construction.

[0104] Optionally, the first instruction includes at least the role that executes the instruction, the role's work content, examples corresponding to the work content, and a detailed description of each part of the knowledge graph, wherein the work content is to convert key-value pair data into a knowledge graph with a graph representation structure, and the examples corresponding to the work content are actual conversion cases for executing the work content.

[0105] It should be noted that the role that executes the instruction refers to the role assigned to the instruction. The work content describes the specific tasks that the role needs to perform, that is, how to convert key-value pair data into a knowledge graph with a graph representation structure, and clarifies the specific requirements and expected results of the conversion task. The examples corresponding to the work content refer to actual conversion cases. These cases show specific examples of how to convert key-value pair data into a knowledge graph with a graph representation structure, which helps the model learn how to correctly perform the conversion task and provide a reference for the model. The given examples should contain all triples of the node type in the knowledge graph. The detailed description is used to describe each part of the knowledge graph, including entity uri, entity namespace, attribute uri, attribute namespace, data type description, RDF and OWL standard guidelines, etc.

[0106] It can be understood that through the above types of instructions, the model can learn how to correctly convert key-value pair data into a knowledge graph, ensuring the accuracy and consistency of the conversion process.

[0107] Step S50, constructing a corpus based on the knowledge graphs corresponding to the data of the multiple original tables.

[0108] It should be noted that step S50 constructs a comprehensive corpus that contains information extracted from multiple knowledge graphs, making the corpus a rich resource that can be used for a variety of purposes, such as natural language processing tasks, machine learning training, data analysis, etc.

[0109] Optionally, the steps of constructing a corpus may include data integration, data cleaning, data standardization, etc.

[0110] It can be understood that step S50 can build a comprehensive corpus based on multiple knowledge graphs, and this corpus can support various applications and services.

[0111] This embodiment obtains original table data, wherein the original table data is data in a structured library table format; maps the original table data according to a preset graph representation structure to obtain mapped original table data; performs format conversion on the mapped original table data to obtain key-value pair data corresponding to the original table data; receives a first instruction input from the outside, inputs the first instruction and the key-value pair data into a preset large model, and obtains a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph of a graph representation structure based on an actual conversion case in the first instruction; constructs a corpus based on the knowledge graphs corresponding to multiple original table data, and is capable of converting structured library tables into a standardized key-value pair format, and then customizes the construction rules of the knowledge graph through external input instructions, thereby generating the required knowledge graph through the large model, realizing automatic conversion from structured library table data to knowledge graph, and constructing a final corpus, effectively improving the automation level of data processing and knowledge graph construction.

[0112] For example, to help understand the implementation process of the corpus generation method obtained by combining this embodiment with the above-mentioned embodiment 1, please refer to Figure 2 , Figure 2 A brief flowchart of a corpus generation method is provided, specifically:

[0113] Get structured multi-source data, which is usually a library table structure. Map the graph representation data structure with the library table structure, and then convert the mapped data to JSON format. Then the user writes prompt large model instructions and uses the large model to convert the library table structure data into graph representation structure data.

[0114] Based on the first embodiment of the present application, in the second embodiment of the present application, the same or similar contents as those in the above-mentioned embodiment 1 can be referred to the above introduction, and will not be repeated in the following. Figure 3 After step S20, the corpus generation method further includes steps B10 to B30:

[0115] Step B10, according to a preset concatenation rule, concatenate each entity of the mapped original table data and the attribute corresponding to the entity into a first character string;

[0116] It should be noted that in the aforementioned steps S30~S40, the model needs to be fine-tuned in advance so that the model can understand the graph representation data structure and can generate a knowledge graph of the graph representation structure directly stored in the database as required according to the description in the prompt large model instruction. At the same time, the context window length of the model needs to cover the nodes, attributes and relationships within five hops of the knowledge graph.

[0117] Optionally, for each entity, prepare an empty string to store the final first string, extract the unique identifier of each entity from the mapped original table data, add each attribute of each entity to the first string according to the preset concatenation rule, and add the entity identifier at the beginning of the first string to clearly know which entity the string represents. After completing the above steps, a complete string is obtained, which contains the information of the entity identifier and all its attributes.

[0118] As an example, the original table data after mapping is: Entity: Customer; CustomerID: 1001, FirstName: John, LastName: Doe, Email: john.doe@example.com. Then the first string generated may be "CustomerID: 1001, FirstName=John, LastName=Doe, Email=john.doe@example.com".

[0119] Step B20, concatenating the entities into a second character string according to the relationship between the entities in the mapped original table data;

[0120] Optionally, for each entity, prepare an empty string to store the final second string, extract the unique identifier (such as ID) of each entity from the mapped original table data, determine the relationship type between each entity and other entities, and for each entity, find out other entities associated with it. Use a preset format to concatenate the identifiers and relationship types of related entities into the second string to obtain a complete string that contains the entity identifier and its related entity identifiers and relationship type information.

[0121] As an example, the original table data after mapping includes: Entity A: Customer CustomerID: 1001; Entity B: Order OrderID: 2001; Entity C: Product ProductID: 3001. Relationships between entities: There is a placed_order relationship between Customer1001 and Order 2001; There is a contains_product relationship between Order 2001 and Product 3001. Then, the generated second string can be: for Customer 1001: CustomerID: 1001: placed_order = OrderID: 2001; for Order 2001: OrderID: 2001: placed_by = CustomerID: 1001, contains_product = ProductID: 3001; for Product 3001: ProductID: 3001: included_in = OrderID: 2001.

[0122] Step B30, generating a knowledge graph according to the first character string and the second character string.

[0123] It should be noted that the first string containing entities and attributes and the second string containing the relationships between entities have been determined. Therefore, the nodes and edges of the knowledge graph can be generated based on the first string and the second string, thereby directly generating the knowledge graph. This method does not require fine-tuning of the model and is suitable for scenarios that do not require deep semantic understanding and complex relationship processing. Although it may not be as powerful as the model-based method in some aspects, it has obvious advantages in simplicity and ease of implementation.

[0124] This embodiment uses preset splicing rules to splice each entity of the mapped original table data and the attributes corresponding to the entity into a first string; splices each entity into a second string according to the relationship between the entities in the mapped original table data; generates a knowledge graph based on the first string and the second string. Through the traditional data text splicing method, structured data can also be converted into a knowledge graph with a graph representation structure without using a large model.

[0125] Based on the first embodiment of the present application, in the third embodiment of the present application, the same or similar contents as those in the first embodiment can be referred to the above introduction, and will not be repeated hereafter. On this basis, after step S10, the corpus generation method can also include steps C10 to C30:

[0126] Step C10, obtaining a fine-tuning dataset, wherein the fine-tuning dataset includes a plurality of historical knowledge graphs and a natural language description of the original table data corresponding to each historical knowledge graph;

[0127] It should be noted that natural language description refers to the use of everyday language (such as Chinese, English, etc.) to describe entities and their relationships in the knowledge graph.

[0128] Step C20, determining a pre-trained model, and fine-tuning the pre-trained model by fine-tuning the data set to obtain a fine-tuned generative model, wherein the generative model is used to parse the second instruction input externally, and generate a knowledge graph corresponding to the natural language description of the original table data;

[0129] It should be noted that the pre-trained model can be a language model based on the Transformer architecture, such as BERT, GPT series or other similar models. You need to choose a model that has been pre-trained on a large amount of general corpus as a basis. The fine-tuned generative model is a model that has been fine-tuned on data in a specific field, which can receive natural language descriptions as input and output a knowledge graph that matches it.

[0130] It can be understood that based on the pre-trained model, fine-tuning is performed through domain-specific data to obtain a generative model that can accurately parse natural language descriptions and generate corresponding knowledge graphs. The fine-tuned model can not only better understand natural language input, but also more accurately generate structured knowledge graph representations.

[0131] Step C30, receiving a second instruction input from the outside, inputting the second instruction into the generation model, and obtaining a knowledge graph output by the generation model, wherein the second instruction includes a natural language description corresponding to the original table data.

[0132] It should be noted that the second instruction refers to a natural language description provided by the user or the system, which can be a description of a concept, entity or relationship. The second instruction usually appears in the form of text, for example: "Please list the founders of Apple and their related information.

[0133] The generative model used has been fine-tuned in step C20 and can better understand the natural language description of a specific field. The received second instruction is input into the generative model for processing. The generative model will parse the natural language description and try to build a knowledge graph corresponding to the description content.

[0134] This embodiment obtains a fine-tuning dataset, wherein the fine-tuning dataset includes multiple historical knowledge graphs and a natural language description of the original table data corresponding to each historical knowledge graph; determines a pre-trained model, fine-tunes the pre-trained model through the fine-tuning dataset, and obtains a fine-tuned generation model, wherein the generation model is used to parse a second instruction input from the outside to generate a knowledge graph corresponding to the natural language description of the original table data; receives a second instruction input from the outside, inputs the second instruction into the generation model, and obtains a knowledge graph output by the generation model, wherein the second instruction includes the natural language description corresponding to the original table data, and by fine-tuning the initial model in advance, the model can understand the graph representation data structure, and can generate a knowledge graph directly stored in the database as required according to the description in the instruction.

[0135] It should be noted that the above examples are only used to understand the present application and do not constitute a limitation on the corpus generation method of the present application. More simple transformations based on this technical concept are all within the scope of protection of the present application.

[0136] This application also provides a corpus generation device, please refer to Figure 4 , the corpus generating device comprises:

[0137] The acquisition module 10 is used to acquire original table data, wherein the original table data is data in a structured library table format;

[0138] A mapping module 20, configured to perform mapping processing on the original table data according to a preset graph representation structure to obtain mapped original table data;

[0139] The conversion module 30 is used to convert the format of the mapped original table data to obtain key-value pair data corresponding to the original table data;

[0140] An execution module 40 is configured to receive a first instruction input from an external source, input the first instruction and the key-value pair data into a preset large model, and obtain a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction;

[0141] The generating module 50 is used to construct a corpus based on the knowledge graphs corresponding to the plurality of original table data.

[0142] The corpus generation device provided by the present application adopts the corpus generation method in the above embodiment, which can solve the technical problem of how to achieve effective conversion of structured library table data to knowledge graph. Compared with the prior art, the beneficial effects of the corpus generation device provided by the present application are the same as the beneficial effects of the corpus generation method provided by the above embodiment, and the other technical features in the corpus generation device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0143] The present application provides a corpus generation device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the corpus generation method in the above-mentioned embodiment 1.

[0144] Reference below Figure 5 , which shows a schematic diagram of the structure of a corpus generation device suitable for implementing the embodiment of the present application. The corpus generation device in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Descriptions), PMPs (Portable Media Players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The corpus generation device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0145] like Figure 5As shown, the corpus generation device may include a processing device 1001 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM: Read Only Memory) 1002 or a program loaded from a storage device 1003 to a random access memory (RAM: Random Access Memory) 1004. In RAM1004, various programs and data required for the operation of the corpus generation device are also stored. The processing device 1001, ROM1002, and RAM1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input device 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output device 1008 including, for example, a liquid crystal display (LCD: Liquid Crystal Display), a speaker, a vibrator, etc.; a storage device 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 1009. The communication device 1009 can allow the corpus generation device to communicate with other devices wirelessly or by wire to exchange data. Although the figure shows a corpus generation device with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems can be implemented or have alternatively.

[0146] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0147] The corpus generation device provided by the present application adopts the corpus generation method in the above embodiment, which can solve the technical problem of how to achieve effective conversion of structured library table data to knowledge graph. Compared with the prior art, the beneficial effects of the corpus generation device provided by the present application are the same as the beneficial effects of the corpus generation method provided by the above embodiment, and the other technical features in the corpus generation device are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0148] It should be understood that the various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0149] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0150] The present application provides a computer-readable storage medium having computer-readable program instructions (ie, computer programs) stored thereon, and the computer-readable program instructions are used to execute the corpus generation method in the above-mentioned embodiment.

[0151] The readable storage medium provided in this application is a computer-readable storage medium, which stores computer-readable program instructions (i.e., computer programs) for executing the above-mentioned corpus generation method, and can solve the technical problem of how to effectively convert structured library table data into knowledge graphs. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as the beneficial effects of the corpus generation method provided in the above-mentioned embodiment, and will not be repeated here.

[0152] The present application also provides a computer program product, including a computer program, which implements the steps of the above-mentioned corpus generation method when executed by a processor.

[0153] The computer program product provided by this application can solve the technical problem of how to effectively convert structured library table data into knowledge graphs. Compared with the prior art, the beneficial effects of the computer program product provided by this application are the same as the beneficial effects of the corpus generation method provided in the above embodiment, which will not be repeated here.

[0154] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A corpus generation method, characterized in that: The corpus generation method comprises: Acquire original table data, wherein the original table data is data in a structured library table format; Performing mapping processing on the original table data according to a preset graph representation structure to obtain mapped original table data; Performing format conversion on the mapped original table data to obtain key-value pair data corresponding to the original table data; Receive a first instruction input from an external source, input the first instruction and the key-value pair data into a preset large model, and obtain a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction, and the first instruction at least includes a role that executes the instruction, the work content of the role, an example corresponding to the work content, and a detailed description of each part of the knowledge graph, wherein the work content is to convert the key-value pair data into a knowledge graph with a graph representation structure, and the example corresponding to the work content is an actual conversion case for executing the work content; Building a corpus based on the knowledge graphs corresponding to the plurality of original table data; The step of converting the format of the mapped original table data to obtain key-value pair data corresponding to the original table data includes: Traverse all entities in the mapped original table data and obtain all attributes corresponding to each of the entities; Using the attribute name corresponding to each of the attributes as a key and each of the attributes as a value, creating a plurality of key-value pairs corresponding to each of the entities; Key-value pair data including all key-value pairs in each of the entities is created according to the multiple key-value pairs corresponding to each of the entities.

2. The corpus generation method according to claim 1, characterized in that: The graph representation structure includes entities, attributes, and relationships between entities. The step of mapping the original table data according to the preset graph representation structure to obtain the mapped original table data includes: Determine the entities, attributes and relationships between entities in the original table data that match the graph representation structure, and map the original table data according to the matching entities, attributes and relationships between entities to obtain mapped original table data.

3. The corpus generation method according to claim 2, characterized in that: The step of determining entities, attributes and relationships between entities in the original table data that match the graph representation structure includes: Determine the entity types corresponding to different types of original table data; Determine each row of data in each of the original table data as an entity of the graph representation structure; Determine the header information of the original table data corresponding to each entity type, determine the attribute name of each entity according to the header information, and extract the attribute value corresponding to each attribute name in each entity from each original table data, and use the attribute name and attribute value corresponding to each row of data in the original table data as the attribute of the corresponding entity; The relationship types between different types of original table data are analyzed, and the relationship between entities is determined according to the relationship types.

4. The corpus generation method according to claim 1, characterized in that: After the step of mapping the original table data according to the preset graph representation structure to obtain the mapped original table data, the method further includes: According to a preset concatenation rule, concatenate each entity of the mapped original table data and the attribute corresponding to the entity into a first string; According to the relationship between the entities in the mapped original table data, the entities are concatenated into a second string; A knowledge graph is generated according to the first character string and the second character string.

5. The corpus generation method according to claim 1, characterized in that: After the step of obtaining the original table data, the method further includes: Obtain a fine-tuning dataset, wherein the fine-tuning dataset includes a plurality of historical knowledge graphs and a natural language description of original table data corresponding to each of the historical knowledge graphs; Determine a pre-trained model, and fine-tune the pre-trained model using the fine-tuning data set to obtain a fine-tuned generative model, wherein the generative model is used to parse the second instruction input externally to generate a knowledge graph corresponding to the natural language description of the original table data; Receive a second instruction input from the outside, input the second instruction into the generation model, and obtain the knowledge graph output by the generation model, wherein the second instruction includes a natural language description corresponding to the original table data.

6. A corpus generation device, characterized in that: The corpus generating device comprises: An acquisition module, used for acquiring original table data, wherein the original table data is data in a structured library table format; A mapping module, used to perform mapping processing on the original table data according to a preset graph representation structure to obtain mapped original table data; A conversion module, used to convert the format of the mapped original table data to obtain key-value pair data corresponding to the original table data; an execution module, configured to receive a first instruction input from an external source, input the first instruction and the key-value pair data into a preset large model, and obtain a knowledge graph output by the preset large model, wherein the preset large model converts the key-value pair data into a knowledge graph with a graph representation structure based on an actual conversion case in the first instruction, wherein the first instruction at least includes a role that executes the instruction, the work content of the role, an example corresponding to the work content, and a detailed description of each part of the knowledge graph, wherein the work content is to convert the key-value pair data into a knowledge graph with a graph representation structure, and the example corresponding to the work content is an actual conversion case for executing the work content; A generation module, used to construct a corpus based on the knowledge graphs corresponding to the plurality of original table data; Wherein, the conversion module is specifically used for: Traverse all entities in the mapped original table data and obtain all attributes corresponding to each of the entities; Using the attribute name corresponding to each of the attributes as a key and each of the attributes as a value, creating a plurality of key-value pairs corresponding to each of the entities; Key-value pair data including all key-value pairs in each of the entities is created according to the multiple key-value pairs corresponding to each of the entities.

7. A corpus generation device, characterized in that: The device comprises: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program is configured to implement the steps of the corpus generation method according to any one of claims 1 to 5.

8. A storage medium, characterized in that: The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, the steps of the corpus generation method according to any one of claims 1 to 5 are implemented.

Images