Knowledge graph generation program, method and system thereof

Abstract

To enable even non-expert users to objectively evaluate and classify publicly available datasets, we provide a structured information map that highly integrates natural language descriptions with analytical facts from real data. [Solution] A first entity 19 that holds property information is generated along with a second property 17 obtained from the analysis of the actual data 11. A prompt 21 defining extraction rules specialized for dataset management is input to the large-scale language model 25, and a first relation 27 and a second entity 29 that indicate the connections between information are extracted. A third entity 31 and a second relation 33 generated from the actual data 11 are generated. All of these entities are treated as nodes, and a multimodal knowledge graph 5 is constructed by connecting them with each relation.

Description

Technical Field

[0001] The present invention relates to a method for generating a multimodal knowledge graph targeting a plurality of datasets, and to a program and a system for executing the method. Background Art

[0002] When machine learning engineers or data analysts who are not experts in the relevant field attempt to access publicly available datasets, it is difficult to objectively evaluate such datasets. Existing metadata and descriptions attached to data often do not sufficiently explain the properties of the data itself. Therefore, there is a need for a technology that can structurally evaluate and classify datasets by comprehensively analyzing the properties of the data itself and metadata, and graphing similarities and relationships between the target dataset and other datasets. Prior Art Documents Patent Documents

[0003] Patent Document 1 Japanese Unexamined Patent Application Publication No. 2026-59981 Summary of the Invention Problems to be Solved by the Invention

[0004] However, the granularity of metadata and description text of publicly available datasets is inconsistent, and it is not easy to cross-sectionally grasp the characteristics of each dataset. Furthermore, existing knowledge graph generation techniques have a problem in that it is difficult to accurately incorporate continuous values such as numerical scores obtained from analysis of actual data as relationships in the graph.

[0005] This invention has been made in view of the problems of the prior art described above, and aims to provide a knowledge graph generation program, method, and system that highly integrates explanations in natural language with the results of analysis of real data, structurally visualizes similarities and relationships with other datasets, and enables objective evaluation. [Means for solving the problem]

[0006] The knowledge graph program of the present invention is a program that causes a computer to generate a knowledge graph for multiple datasets, and causes the computer to execute the following: a first step of acquiring a dataset that includes actual data and a first property that indicates the attributes of the actual data; a second step of generating a first entity with properties based on the first property and a second property obtained by analyzing the actual data; a third step of generating a prompt using the first entity; a fourth step of inputting the prompt generated in the third step into a large-scale language model and generating a first relation and a second entity by extracting the relationships with other entities; and a fifth step of generating a third entity by making the second property into an entity.

[0007] In this configuration, the second entity is generated by inputting the second property, which is the result of analyzing real data, and the first entity, which is generated from the first property, into a large-scale language model. Based on the second property, a second relationship and a third entity are generated. This enables the generation of a multimodal knowledge graph in which natural language explanations and analytical facts from real data are highly integrated.

[0008] Preferably, the fifth step generates a second relation based on the actual data. This configuration allows for the direct generation of a second relationship based on the analysis of real data, bypassing large-scale language models. This enables the accurate representation of relationships, such as numerical scores, which large-scale language models struggle with, in the knowledge graph.

[0009] Preferably, the third step aggregates the multiple first entities and generates prompts for a large language model that define extraction rules specifically for extracting and structuring datasets. This configuration allows for the efficient generation of a knowledge graph with a highly accurate structure optimized for dataset management by aggregating multiple first entities and then instructing a large-scale language model with extraction rules specifically tailored for dataset extraction and structuring.

[0010] Preferably, the first property is one of the following pieces of information associated with the dataset: a description, an external reference, or the data format. This configuration leverages existing descriptions, external references, or data formats associated with the dataset as primary properties, appropriately reflecting the dataset's background information and formal characteristics as elements of the knowledge graph, and enabling multifaceted exploration.

[0011] Preferably, the fifth step involves performing at least one of the following on the actual data: feature extraction using machine learning, evaluation, or statistical analysis, and generating the second relation based on the evaluation score obtained, data variability due to clustering, annotation deficiencies, or attribute information of the data structure. This configuration allows for the creation of a knowledge graph that can objectively evaluate the quality and reliability of a dataset by performing feature extraction and statistical analysis on real data using machine learning, and then generating a second relationship based on objective indicators such as the obtained evaluation score, data variability, and annotation deficiencies.

[0012] Preferably, the second step involves performing at least one of the following on the actual data: feature extraction using machine learning, evaluation, or statistical analysis, and generating the second property based on the evaluation score obtained, data variability due to clustering, annotation deficiencies, or attribute information of the data structure. This configuration allows for the generation of objective indicators obtained from analyzing real data as a second property, which is then incorporated into the first entity. This makes it possible to accurately convey the specific properties of the real data to a large-scale language model.

[0013] Preferably, the computer is further instructed to perform a sixth step of matching the second entity with the first relation and the third entity with the second relation. This configuration ensures the accuracy of the information contained in the knowledge graph and constructs a consistent graph structure by performing a sixth step that reconciles the information generated by the large-scale language model with the information generated based on the analysis of real data.

[0014] Preferably, the fourth step generates the first relation and the second entity using the large-scale language model. This configuration allows for the dynamic generation of a first relation with complex relationships to other entities, and a second entity exhibiting new attributes, based on extraction rules defined in the prompt, by using a large-scale language model.

[0015] Preferably, the actual data includes images, time-series data, and annotations or labels contained therein. This configuration treats images, time-series data, and associated labels as real data, allowing for the incorporation of data distribution characteristics and physical features that cannot be expressed through text information alone into the knowledge graph generation process, thus enabling multimodal analysis.

[0016] The knowledge graph generation method of the present invention is executed by a computer, the method comprising: a first step of acquiring a dataset including actual data and a first property indicating an attribute of the actual data; a second step of generating a first entity with a property based on the first property and a second property obtained by analyzing the actual data; a third step of generating a prompt using the first entity; a fourth step of inputting the prompt generated in the third step into a large-scale language model, and generating a first relation obtained by extracting relevance with other entities and a second entity; and a fifth step of entityizing the second property to generate a third entity.

[0017] The knowledge graph generation system of the present invention comprises: a first means for acquiring a dataset including actual data and a first property indicating an attribute of the actual data; a second means for generating a first entity with a property based on the first property and a second property obtained by analyzing the actual data; a third means for generating a prompt using the first entity; a fourth means for inputting the prompt generated by the third means into a large-scale language model, and generating a first relation obtained by extracting relevance with other entities and a second entity; and a fifth means for entityizing the second property to generate a third entity. [Effects of the Invention]

[0018] According to the present invention, it is possible to provide a knowledge graph generation program, a method and a system thereof that highly integrate natural language descriptions and analysis results of actual data, structurally visualize similarities and relationships with other datasets, and enable objective evaluation. [Brief Description of the Drawings]

[0019] [Figure 1] It is a functional block diagram of the knowledge graph generation system 3. [Figure 2] It is a conceptual diagram of knowledge graph generation. [Figure 3] It is a diagram for explaining an example of the knowledge graph 5 shown in FIG. 1. [Figure 4] It is a flowchart for explaining the operation of the knowledge graph generation system 3. [Figure 5] It is a diagram for explaining the structure of a general knowledge graph. [Figure 6] It is a diagram for explaining an example of the knowledge graph 5 generated by the knowledge graph generation system 3 according to the embodiment of the present invention. [Figure 7] It is a diagram for explaining an example of a prompt used in general knowledge graph generation. [Figure 8] It is a diagram for explaining an example of the prompt 21 generated by the knowledge graph generation system 3 according to the embodiment of the present invention. MODE FOR CARRYING OUT THE INVENTION

[0020] Hereinafter, a knowledge graph generation system according to an embodiment of the present invention will be described. <Knowledge Graph Generation System 3> FIG. 1 is a functional block diagram of the knowledge graph generation system 3. FIG. 2 is a conceptual diagram of a method for generating the knowledge graph 5 by the knowledge graph generation system 3 shown in FIG. 1. FIG. 3 is a diagram for explaining an example of the knowledge graph 5 generated by the knowledge graph generation system 3. As shown in FIG. 1, the knowledge graph generation system 3 includes, for example, a communication unit 75, an input unit 77, a memory 79, and a processing unit 81.

[0021] The communication unit 75 communicates with an external system. The input / output unit 77 is a terminal or the like for inputting data from the outside. The memory 79 stores programs executed by the processing unit 81. The processing unit 81 executes the program PRG stored in the memory 79 to perform processing of the knowledge graph generation system 3 defined in the present embodiment.

[0022] The processing unit 81 performs a first step of acquiring multiple datasets 2, which include actual data 11 containing images or time-series data, and first properties 15 indicating the attributes of the actual data 11. Here, the first properties 15 are information such as descriptions, external references, and data formats associated with the datasets 2.

[0023] The processing unit 81 performs a second step of generating a first entity 19 that holds property information, based on the first property 15 and the second property 17 obtained by analyzing the actual data 11.

[0024] The processing unit 81 aggregates multiple first entities 19 and performs a third step of generating a prompt 21 for the large-scale language model 25 that defines extraction rules specifically for extracting and structuring the dataset. In this process, the large-scale language model LLM25 is instructed to classify the extraction targets using specific labels such as dataset, source, category, or task, to use a fixed unique identifier that includes the dataset name, and to extract the dependencies between categories as relationships.

[0025] This extraction rule includes the following constraints: Firstly, it includes instructions to strictly assign dataset-specific attribute labels such as source, category, and task to the nodes to be extracted, with the dataset as the primary focus. Secondly, if a dependency relationship is recognized between properties, such as category and subcategory types, it includes instructions to extract that hierarchical structure as a relation. These dataset-specific constraints enable highly accurate structuring, which is difficult to achieve with general-purpose knowledge graph generation.

[0026] The processing unit 81 inputs the prompt 21 generated in the third step into the large-scale language model 25 and executes a fourth step in which it generates a first relation 27 that extracts the relationships with other entities and a second entity 29.

[0027] The processing unit 81 performs at least one of the following on the actual data 11: feature extraction using machine learning, evaluation, or statistical analysis. Based on the evaluation score, data variability, annotation deficiencies, or data structure attribute information obtained as a result, it executes a fifth step in which it creates a second property 17 into an entity and generates a third entity 31. Here, the actual data 11 includes all data formats, such as videos, 3D data, and tabular data, as well as annotations and labels for them.

[0028] Furthermore, based on the information obtained from the properties that show the analysis results, a second relation 33 is generated by setting up a rule-based relation that does not rely on a large-scale language model 25. By using the second relation 33, continuous values ​​(such as scores), which LLM struggles with, can be directly defined using a rule-based approach based on accurate analysis results. While large-scale language models excel at understanding the context of natural language, they carry the risk of hallucination (misinformation) when accurately comparing continuous values ​​such as numerical scores or defining relationships based on rigorous facts. In this configuration, accurate numerical values ​​definitively derived during the analysis of real data are directly injected into the graph using a rule-based approach. This enables highly accurate relationship assignment based on quantitative criteria, which has been difficult to support with existing knowledge graph generation tools.

[0029] The processing unit 81 executes a sixth step in which it generates a knowledge graph 5 by using the first entity 19, the second entity 29, and the third entity 31 as nodes and linking each node using the first relation 27 and the second relation 33.

[0030] <Dataset> While there are no fundamental restrictions on the datasets handled by Knowledge Graph Generation System 3, it is desirable to define a certain range and theme depending on the purpose. This serves the purpose of defining the concept and use of the knowledge graph to be created, but it is also a necessary selection process to ensure that the evaluations of the dataset during the subsequent entity creation process are comparable.

[0031] Examples are given below. Objective: To collect data from other fields that is suitable for creating an anomaly detection model for motor vibration data. Scope: Handles time-series sensor data datasets (does not handle image data or non-sensor count data).

[0032] Objective: To collect data for the detection and segmentation of specific microorganisms. Scope: This dataset handles microscopic images of microorganisms (it does not handle RGB images, people, landscapes, etc.).

[0033] The following information is available in publicly available open datasets, but not all datasets display this information. - Dataset name - URL - Source - Field (e.g., medical, industrial) - Target (EEG, temperature, electrical current, movement pathway, etc.) - Tasks (anomaly detection, prediction, etc.) - Purpose of acquisition, background - License - Related papers

[0034] Furthermore, the dataset contains the data itself. From this, we can also obtain the following information: - Number and types of variables - Number of data - Annotation information This information is difficult to handle as tabular data because it is not necessarily displayed in the same quantity or with the same granularity across all datasets.

[0035] The information contained in a dataset includes both information that can be used directly as property information and information that requires some kind of processing. Because the Knowledge Graph (KG) allows for the addition of any feature as an element in its data structure, this "some kind of processing" can be extended. Because datasets contain real data internally, it is possible to perform machine learning-like feature extraction using this data, including supervised and unsupervised learning and evaluation. By combining these elements, the input in this generation process will have two modes: "a description of the dataset in natural language" and "actual data such as images and time series," enabling the generation of a knowledge graph containing multimodal information.

[0036] <Create a dataset entity> In this embodiment, an entity is a node in the knowledge graph 5. To create a Knowledge Graph 5 centered on datasets, the datasets must be reliably represented as entities. When specifying which information to consider as entities in LLM25, the entities in the dataset must be created beforehand.

[0037] Furthermore, by assigning information associated with the dataset as properties to this entity, LLM25 recognizes that the dataset and property information are strongly linked. The way properties are assigned affects the accuracy of Knowledge Graph 5 generation, and can be done, for example, as follows.

[0038] 1. Specify properties using natural language. 2. Assign properties using structured data. 3. Combining natural language and structured data

[0039] Furthermore, there are various formats for evaluating and analyzing the data itself, as well as for presenting the results. The following are some methods for evaluation and analysis.

[0040] 1. Evaluate the dataset in relation to its intended use, which is the task it is used for. 2. Assign the same task (such as a classification problem) to each dataset and evaluate its performance. 3. Perform data segmentation and quantify the variability using clustering or similar methods. 4. Apply a dataset evaluation method such as ConfidentLearning to obtain a score.

[0041] For example, if you collect image classification datasets with the intention of using them for anomaly detection, you can use metrics such as the accuracy of multi-class classification, the variance of the distribution based on feature distance, and the number of mislabeled items detected in the dataset, which were the original intended uses of the dataset. When time-series sensor data is collected for fault prediction, the prediction accuracy when actually used for fault prediction, the detection accuracy when used for fault detection rather than prediction, the frequency of fault occurrence, and the waveform type during normal operation can be used. Furthermore, a key feature of this technology is its ability to incorporate freely extended metrics, not limited to those listed here. Furthermore, there are several possible ways to present these results.

[0042] 1. Provide qualitative evaluations in natural language. 2. Provide a summary of the evaluation results in natural language. 3. Provide a summary of the evaluation results as structured data. 4. Provide detailed analysis results as structured data.

[0043] Knowledge graph generation system 3 creates dataset entities including these elements and performs performance evaluation.

[0044] The operation of the knowledge graph generation system 3 will be explained below based on a flowchart. Figure 4 is a flowchart illustrating the operation of the knowledge graph generation system 3 according to an embodiment of the present invention. Step ST11: The processing unit 81 of the knowledge graph generation system 3 acquires multiple datasets 2, each containing actual data 11, which includes images or time-series data, and a first property 15, which indicates the attributes of the actual data 11. (First step) The first property 15 is information associated with dataset 2, such as descriptions, external references, and data format.

[0045] Step ST12: The processing unit 81 generates a first entity (property-attached entity) 19 that holds property information, based on the first property 15 and the second property 17 obtained by analyzing the actual data 11. (Second step) Furthermore, the processing unit 81 performs at least one of the following on the actual data: feature extraction using machine learning, evaluation, or statistical analysis. Based on the resulting evaluation score, data variability due to clustering, annotation deficiencies, or attribute information of the data structure, it generates a second property.

[0046] Step ST13: The processing unit 81 aggregates multiple first entities 19 and generates a prompt 21 for the large-scale language model 25 that defines extraction rules specifically for extracting and structuring the dataset. (Third step) In this process, the large-scale language model 25 is instructed by prompt 21 to classify the extraction targets using specific labels such as dataset, source, category, or task, to use a fixed unique identifier that includes the dataset name, and to extract the dependencies between categories as relationships.

[0047] Step ST14: The processing unit 81 inputs the prompt 21 generated in step ST13 into the large-scale language model 25 and generates a first relation 27, which extracts the relationships with other entities, and a second entity 29. (Fourth step)

[0048] Step ST15: The processing unit 81 creates a third entity 31 by converting the second property 17 into an entity. (Step 5) Furthermore, the processing unit 81 generates a second relation 33 based on information obtained from properties indicating the analysis results, using rule-based relation setting that does not rely on the large-scale language model 25. This enables direct relation setting based on accurate analysis results. As described above, a knowledge graph is obtained by generating each entity and each relationship.

[0049] Step ST16: The processing unit 81 performs a process to ensure consistency between the second entity, the first relation, and the third entity and the second relation (data analysis result). Furthermore, at this stage, the processing unit 81 performs post-processing to detect and remove obvious errors in the generated knowledge graph, or to redo the process. For example, if the same result is included in the knowledge graph, it adjusts it by prioritizing the one generated later.

[0050] Figure 5 is a diagram illustrating a typical knowledge graph. As shown in Figure 5, the created KG uses the dataset as the main axis of entities, and entities are formed from the related fields, related organizations, and people within the dataset, as well as the labels, column names, licenses, and even evaluation and analysis results contained within the dataset.

[0051] Figure 6 is a diagram illustrating an example of a knowledge graph 5 generated by a knowledge graph generation system 3 according to an embodiment of the present invention. Figure 7 illustrates an example of prompts used in typical knowledge graph generation. Figure 8 is a diagram illustrating an example of a prompt 21 generated by the knowledge graph generation system 3 of an embodiment of the present invention.

[0052] In this embodiment, new entities can be added at any time to extend the knowledge graph 5. The user can issue several commands to Knowledge Graph 5. For example, consider the following command: - Search for datasets in field B related to person A. - List relevant information about dataset C. - Present the dataset most similar to dataset D, and extract the graph structure used to determine its similarity. - For the newly added dataset E, estimate which field it belongs to. - Estimate the evaluation of the newly added dataset F on a criterion called G. These commands follow the general input format of Knowledge Graph 5, but since techniques for commands using natural language also exist, using them allows for more flexible commands.

[0053] Knowledge graph generation system 3 aggregates the created property entities and generates prompts based on its own rules. The entity with properties itself is entered into these LLMs as a prompt, but depending on the algorithm (such as LightRAG), it may also be provided as a RAG (Retrieval-Augmented Generation).

[0054] According to the Knowledge Graph Generation System 3, even non-expert machine learning engineers and data analysts can objectively evaluate and classify publicly available datasets through the Knowledge Graph 5. By highly integrating descriptive text in natural language with objective indicators obtained from the analysis of real data, it becomes possible to structurally grasp the nature of the data itself, which cannot be determined by metadata alone.

[0055] Knowledge Graph Generation System 3 can directly establish rule-based relationships for continuous values ​​(numerical data) such as evaluation scores, which are difficult for large-scale language models to handle, based on accurate facts derived during the analysis phase. This dramatically improves the accuracy of "extracting similarities based on accuracy scores" and "data exploration based on quantitative criteria," which were difficult with existing knowledge graph generation tools.

[0056] According to the Knowledge Graph Generation System 3, by leveraging the structure of the knowledge graph, it is possible to flexibly search for datasets related to specific individuals or organizations, suggest similar datasets, and even estimate the attributes and evaluations of newly added datasets. This allows for the immediate identification of the most suitable data for a given purpose from a vast collection of datasets, thereby accelerating decision-making in data utilization.

[0057] The present invention is not limited to the embodiments described above. In other words, those skilled in the art may make various modifications, combinations, subcombinations, and substitutions with respect to the components of the embodiments described above, within the technical scope of the present invention or its equivalents.

[0058] When generating prompts, if information about property-bearing entities is input into a large-scale language model, depending on the characteristics of the algorithm (e.g., LightRAG), it may be possible to input the information directly as a prompt or to provide the information as RAG (Retrieval-Augmented Generation).

[0059] Furthermore, the "processing" performed on the actual data during the generation of the second property can be expanded according to the purpose of the dataset. For example, for an anomaly detection dataset, the variance of the distribution based on feature distance can be extracted, and for fault prediction, the waveform type during normal operation can be extracted. Analysis methods can be arbitrarily added to match the nature of the target data.

[0060] The knowledge graph can be expanded at any time, allowing for the dynamic addition of new datasets and analysis results discovered during operation without compromising the existing graph structure, thus continuously growing the knowledge map. Furthermore, user commands can be input via structured query languages ​​(such as SPARQL) in addition to natural language processing techniques, enabling more flexible interactive operation. [Industrial applicability]

[0061] The present invention is applicable to knowledge graph generation systems. [Explanation of symbols]

[0062] 2…Dataset 3…Knowledge graph generation system 5…Knowledge Graph 11…Actual data 15…First property 17…Second property 19…The first entity (entity with properties) 21…Prompt 25…Large-scale language models (LLMs) 27…The first relation (edge) 29... The second entity 31... The Third Entity 33…Second Relationship 75... Communications Department 77...Input / output section, 79…Memory 81… Processing Section

Claims

1. A program that causes a computer to generate a knowledge graph from multiple datasets, A first step of obtaining a dataset that includes actual data and a first property that indicates the attributes of the actual data, A second step of generating a first entity with properties based on the first property and the second property obtained by analyzing the actual data, A third step of generating a prompt using the first entity, The prompt generated in the third step is input into a large-scale language model, and a fourth step is generated in which a first relation is obtained by extracting the relationships with other entities, and a second entity is generated. A fifth step of generating a third entity by making the second property described above into an entity, A knowledge graph generation program that causes a computer to execute this process.

2. The fifth step generates a second relationship based on the actual data. A knowledge graph generation program according to claim 1.

3. The third step aggregates the multiple first entities and generates prompts for the large-scale language model that define extraction rules specifically for extracting and structuring the dataset. A knowledge graph generation program according to claim 1.

4. The first property is one of the following pieces of information associated with the dataset: a description, an external reference, or the data format. The knowledge graph generation program according to claim 3.

5. The fifth step involves performing at least one of the following on the actual data: feature extraction using machine learning, evaluation, or statistical analysis, and generating the second relation based on the evaluation score obtained, data variability due to clustering, annotation deficiencies, or attribute information of the data structure. The knowledge graph generation program according to claim 2.

6. The second step involves performing at least one of the following on the actual data: feature extraction using machine learning, evaluation, or statistical analysis, and generating the second property based on the resulting evaluation score, data variability due to clustering, annotation deficiencies, or attribute information of the data structure. A knowledge graph generation program according to claim 1.

7. A sixth step is to perform a process to ensure consistency between the second entity and the first relation and between the third entity and the second relation. The knowledge graph generation program according to claim 2, which causes the computer to further execute the following.

8. The fourth step involves generating the first relation and the second entity using the large-scale language model. A knowledge graph generation program according to claim 1.

9. The aforementioned actual data includes images, time-series data, and annotations or labels contained therein. A knowledge graph generation program according to claim 1.

10. A method for generating a knowledge graph using multiple datasets, A first step of obtaining a dataset that includes actual data and a first property that indicates the attributes of the actual data, A second step of generating a first entity with properties based on the first property and the second property obtained by analyzing the actual data, A third step of generating a prompt using the first entity, The prompt generated in the third step is input into a large-scale language model, and a fourth step is generated in which a first relation is obtained by extracting the relationships with other entities, and a second entity is generated. A fifth step of generating a third entity by making the second property described above into an entity, A method for generating a knowledge graph that a computer performs.

11. A system for generating knowledge graphs from multiple datasets, A first means for obtaining a dataset including actual data and a first property indicating the attributes of the actual data, A second means for generating a first entity with properties based on the first property and a second property obtained by analyzing the actual data, A third means for generating a prompt using the first entity, The prompt generated by the third means is input into a large-scale language model, and a fourth means generates a first relation obtained by extracting relationships with other entities, and a second entity. A fifth means for generating a third entity by making the second property into an entity, A knowledge graph generation system having [a certain feature].

Images