Multi-source cross-domain data query method and system

Through the multi-source knowledge fusion and query method based on the large language model, homologous knowledge line graphs are constructed and trusted candidate subgraphs are screened, and the problem of multi-source cross-domain data processing is solved, and accurate and efficient query of multi-source cross-domain data is realized, and data processing efficiency and accuracy are improved.

CN119691003AActive Publication Date: 2025-03-25NANJING UNIV OF AERONAUTICS & ASTRONAUTICS

Patent Information

Application Number
CN202510206906.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-02-25
Publication Date
2025-03-25
Estimated Expiration
2045-02-25

AI Technical Summary

Technical Problem

The existing technology is difficult to effectively process multi-source cross-domain data, especially in training large language models. The complex dependencies of multi-modal data reduce the inference performance, and data security, privacy protection and standardization are difficult to solve, and traditional methods are difficult to meet the needs of data governance.

Method used

A multi-source knowledge fusion and query method based on a large language model is proposed. By retrieving multi-source cross-domain data from different sources, a multi-source knowledge graph is constructed, and homologous knowledge line graph is formed through reconstruction and aggregation. The confidence score and source confidence threshold calculation of query nodes are used for filtering trusted candidate sub-graphs, and accurate data query is used to use vertical large models.

Benefits of technology

It realizes accurate and efficient query of multi-source cross-domain data, significantly improves data processing efficiency, reduces processing time costs, and provides more accurate and reliable data results, suitable for a wider range of and complex data scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119691003A_ABST
    Figure CN119691003A_ABST
Patent Text Reader

Abstract

The invention belongs to the technical field of information retrieval, and discloses a multi-source cross-domain data query method and system. The method comprises the following steps: firstly, carrying out data identification, extraction and classification by utilizing a large language model, and constructing a multi-source knowledge graph; secondly, performing deep aggregation and reconstruction of multi-source cross-domain data based on a large language model to obtain a homologous knowledge line graph; thirdly, calculating the confidence degree of the query nodes and the subordinate sub-graphs according to query requirements, rejecting low-quality nodes, and extracting credible candidate sub-graphs from the knowledge graph so as to ensure that a multi-domain retrieval query result is effective; and finally, embedding the screened credible candidate sub-graphs into a cue word template, and carrying out data query by utilizing a vertical large model. According to the multi-source knowledge fusion and query method based on the large language model, cross-domain data in different sources are integrated to form domain knowledge, and accurate and efficient query of the multi-source cross-domain data is achieved through the vertical large model.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of information retrieval, and relates to a multi-source cross-domain data query method and system. Background Art

[0002] With the rapid development of information technology, the digitalization process of various industries has accelerated, and data has exploded. The processing of multi-source cross-domain data has become a key challenge. Multi-source cross-domain data originates from different industries or fields, and its data models, structures, semantics and quality standards vary greatly. It has a wide range of sources, different quality, decentralized management and heterogeneous forms. Although there has been research progress on multi-source cross-domain data, most of them rely on expert engines and rule systems, or are modeled as semantic matching problems. These methods rely on a large amount of labeled data or expert knowledge and have obvious limitations. Faced with the diversified data specifications and standards of thousands of industries, multimodal and multi-level data formats, and massive high-quality data requirements, traditional methods are difficult to meet the current data governance needs. Especially when training vertical large language models, processing multimodal data is crucial, and the complex dependencies between multi-source cross-domain data will reduce the reasoning performance of large language models, and data security, privacy protection and normalization are also urgent issues to be solved, all of which seriously restrict the full mining of data value. In addition, large language models, with their powerful natural language processing capabilities, can extract valuable insights from heterogeneous data sources, promote the effective use of data, and assist in decision-making and business optimization. As an important achievement of knowledge engineering in the era of big data, knowledge graph provides an effective way for the knowledge organization and intelligent application of multi-source cross-domain data. It can process complex unstructured cross-domain multimodal data, generate high-quality structured data graphs, alleviate the problems of hallucination, catastrophic forgetting and incorrect knowledge editing of large language models, enhance data standardization and interpretability, broaden application scenarios and improve governance efficiency. Summary of the invention

[0003] The purpose of the present invention is to propose a multi-source cross-domain data query method, which proposes a multi-source knowledge fusion and query method based on a large language model, integrates cross-domain data from different sources to form domain knowledge, and thus uses a vertical large model to achieve accurate and efficient query of multi-source cross-domain data.

[0004] In order to achieve the above object, the present invention adopts the following technical scheme: A multi-source cross-domain data query method comprises the following steps: Step 1. Retrieve multi-source cross-domain data from different sources and storage forms in the field, and use a large language model to identify, extract and classify the data to build a multi-source knowledge graph; Step 2. Reconstruct and aggregate the multi-source knowledge graph, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph, and obtain a homologous knowledge line graph; Step 3. Based on the query requirements, use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the credible candidate subgraphs related to the query; Step 4. Embed the core content and key related information of the selected credible candidate subgraphs into the prompt word template, and then use the vertical large model to select the data results that highly match the query intent as the data query results.

[0005] In addition, based on the above multi-source cross-domain data query method, the present invention also proposes a corresponding multi-source cross-domain data query system, which adopts the following technical solutions: A multi-source cross-domain data query system includes the following modules: The multi-source knowledge graph construction module is used to retrieve multi-source cross-domain data from different sources and storage forms in the field, and use the large language model to identify, extract and classify the data to build a multi-source knowledge graph; A homologous knowledge line graph construction module is used to reconstruct and aggregate the homologous knowledge line graph obtained, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph, and obtain a homologous knowledge line graph; The trusted candidate subgraph screening module is used to use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the trusted candidate subgraphs related to the query; And a data query module, which is used to embed the core content and key related information of the screened credible candidate subgraphs into the prompt word template, and then use the vertical large model to screen out data results that highly match the query intent.

[0006] The present invention has the following advantages: As described above, the present invention relates to a multi-source cross-domain data query method and system based on the collaboration of domain knowledge and vertical large models. In terms of data processing efficiency, the present invention proposes a homologous knowledge line graph structure, which greatly optimizes the data storage and query architecture. Compared with the traditional method that relies on expert engines and rule systems and requires a large amount of labeled data, it can locate and obtain the required data information more quickly, significantly improve the overall efficiency of data processing, and reduce processing time costs. In terms of data accuracy, the present invention integrates multi-dimensional factors to calculate the confidence of entity nodes and eliminates untrustworthy nodes, effectively eliminating the interference of errors and low-quality data during data query and screening, thereby providing more accurate and reliable candidate subgraphs and data results. In dealing with complex data relationships, the present invention makes full use of the advanced capabilities of large language models and vertical large models to effectively handle the complex dependencies between multi-source cross-domain data, overcome the problem that the existing technology causes loss of large language model reasoning performance when processing such data, and ensures the stability and reliability of data processing, so that it can be applied to a wider range of more complex data scenarios. The method of the present invention improves the query performance and speed of multi-source cross-domain data by utilizing a large language model combined with knowledge graph optimization construction, and at the same time utilizes a vertical large model to perform accurate and efficient data query on data in specific fields. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 Schematic diagram of a multi-source cross-domain data query method according to an embodiment of the present invention; Figure 2 Schematic diagram of the transformation of a linear knowledge graph in a multi-source cross-domain data query method in an embodiment of the present invention; Figure 3 A flowchart of constructing a homologous knowledge line graph in an embodiment of the present invention; Figure 4 Schematic diagram of the process of the trusted candidate subgraph matching method in an embodiment of the present invention; Figure 5 The figure is a flow chart of a multi-source cross-domain data query method in an embodiment of the present invention. DETAILED DESCRIPTION

[0008] The present invention is further described in detail below with reference to the accompanying drawings and specific embodiments:

[0009] Example 1 This embodiment 1 describes a multi-source cross-domain data query method, which uses a large language model to deeply aggregate multi-source cross-domain data in a domain to reconstruct a dense homologous knowledge line graph; and performs credibility analysis and screening on the homologous knowledge line graph to match and obtain credible candidate subgraphs; finally, the screened credible candidate subgraphs are returned to the vertical large model, thereby realizing efficient and accurate data query functions.

[0010] like Figure 1 As shown, this embodiment describes a multi-source cross-domain data query method based on domain knowledge and vertical big model collaboration, which uses multi-source cross-domain knowledge fusion and vertical big model to query multi-source cross-domain data, and the steps are as follows: Step 1. Retrieve multi-source cross-domain data from different sources and storage forms in the search field, and use a large language model to identify, extract, and classify the data to build a multi-source knowledge graph.

[0011] Specifically, we first obtain multi-source cross-domain data from different data sources, integrate multi-source data, and build a knowledge graph. Multi-source cross-domain data types include structured tables, semi-structured XML documents, and unstructured text.

[0012] Use the large language model to extract entities and relationships, and use the general semantic understanding and analysis capabilities of the large language model to deeply identify key information, sort out entity relationships, and obtain entities and relationships between entities in multi-source cross-domain data.

[0013] Use a unified data storage method to initially integrate multi-source cross-domain data. First, design a data conversion parser for different data formats to integrate multi-source cross-domain data and unify its storage format. For actual application scenarios, directly obtain data from multiple data formats of different sources and convert it into a unified standardized representation.

[0014] Specifically, the LLMGraphTransformer library in the Python programming language is used to extract entities and relationships from multi-source knowledge graphs. With its powerful natural language processing capabilities, the library can perform in-depth semantic analysis on data from different data sources, accurately identify key elements such as people, places, events, and things, and simultaneously extract the relationships between them, such as causal relationships, belonging relationships, and association relationships. After that, the file name and file metadata of the data are collected, and the domain where the file is located is divided. The data content is then parsed and stored in the JSON-LD format and converted into linked data. Finally, the present invention assigns a unique identifier to the data to form standardized data.

[0015] In step 1, the process of preliminary fusion of multi-source cross-domain data is as follows: Different data conversion parsers are used to perform data analysis on each different data type, and the analysis of structured data, semi-structured data and unstructured data is distinguished during data analysis.

[0016] For structured data, its parsing is equivalent to storing table data in JSON format. The attribute variables in the file are stored using the column storage model (decomposition storage model, DSM) so that all attribute information can be extracted using column indexes for consistency verification. For semi-structured data, its parsing is equivalent to storing tree data in JSON format. The file is stored in a multi-layer nested JSON format. There is no column index and fast retrieval for data in this format. Tree / graph retrieval algorithms are used for retrieval (such as DFS). For unstructured data, the unstructured data is text data, and the content is stored directly. In the future, LLM is considered for entity / relationship extraction tasks to obtain the corresponding information. Finally, the fusion of multi-source cross-domain data is expressed as the following formula: .

[0017] in, represents the fused multi-source cross-domain data, Represents the data conversion parser for structured data, semi-structured data and unstructured data respectively, A data set that represents structured data, semi-structured data, and unstructured data respectively.

[0018] The present invention is based on a multi-source cross-domain data extraction and storage method to obtain a multi-source knowledge graph architecture, which can provide basic support for the optimization of subsequent step 2 and the query of step 4.

[0019] Step 2. Reconstruct and aggregate the multi-source knowledge graph obtained in step 1, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph to achieve a denser storage structure and improve query efficiency. After reconstructing and aggregating the multi-source knowledge graph, a dense homologous knowledge graph is obtained, which is defined as a homologous knowledge line graph, such as Figure 2 shown.

[0020] Specifically, first, the entities and relationships extracted in step 1 are used to match and align the same or similar entities in different data sources through a text similarity model, and the relationships between entities are sorted and fused to obtain an aggregated homologous data group; then, a homologous knowledge line graph is constructed based on the homologous data group, where each node in the homologous knowledge line graph represents a triple, and the associated edges represent the relationship between triples.

[0021] Based on the multi-source knowledge graph initially constructed in step 1, it is necessary to traverse all graphs to obtain all cross-domain homologous data group sets SVs and all isolated point sets , and finally complete the aggregation and reconstruction of the knowledge graph.

[0022] like Figure 3As shown in the figure, the construction process of the homology knowledge line graph is as follows: Step 2.1. Initialize the unvisited node set, homologous data set, and isolated point set, including initializing the unvisited node set , homologous data set SVs , isolated point set .

[0023] This initialization process lays the foundation for subsequent traversal and data grouping.

[0024] Step 2.2. Traverse each node in the unvisited node set, and for the currently traversed node A, initialize the set of homologous nodes, the set of edges, and related variables of the homologous nodes.

[0025] Traverse the other nodes B in the unvisited node set except the current node A again. When it is determined that the current node A and other nodes B meet the homology relationship, perform the following operations: Add the current node A and other nodes B that satisfy the homology relationship to the homology node set respectively and edge sets ; and remove the current node A and other nodes B from the unvisited node set to avoid repeated access and processing.

[0026] Step 2.3. After determining the homologous nodes and edge sets of a node, make the following judgments: If the same source node set If it is an empty set, add the current node A to the isolated point set In the data set, it is managed separately as a special data point independent of the homologous data group; If the same source node set is not an empty set, then And the corresponding edge set As a homogenous data set , added to the homologous data set In order to conduct unified analysis and processing of homologous data in the future.

[0027] Step 2.4. Repeat steps 2.2 and 2.3 until all nodes in the unvisited node set have been traversed and processed, completing the construction process of the entire homologous knowledge graph, and finally outputting the homologous data set SVs and the isolated point set .

[0028] Step 2.5. For homogeneous data sets Each homologous data set in , which contains the set of homologous nodes and homologous edge sets Constructing homologous linear subgraphs ; All homologous linear subgraphs and isolated point sets Aggregate to get a homologous knowledge line graph .

[0029] It should be noted that the homology knowledge line diagram in the present invention It is only used for confidence calculation and retrieval query of homologous data. Queries for other types of data are still performed in the multi-source knowledge graph in step 1.

[0030] The present invention aggregates data based on the text similarity model, that is, by deeply analyzing the nodes and edges in the knowledge graph, it identifies groups of nodes with similar semantic features and potential associations. These scattered similar semantic nodes are merged and reorganized using text similarity calculation to form a denser storage mode to ensure the efficiency of subsequent query and screening.

[0031] Step 3. Based on the query requirements, use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the credible candidate subgraphs related to the query.

[0032] Specifically, the query demand is parsed into a retrieval instruction, and the homologous knowledge line graph is traversed according to the confidence of the node to screen and match the query-related subgraphs. The obtained subgraph is called a credible candidate subgraph; the confidence of the entity node is calculated by the confidence calculation method of mutual information entropy, and the reliability of the node source, the authority of the node and the historical authority are comprehensively considered, and the probability statistical model and the large language model are used to calculate the score. The present invention introduces a confidence calculation method based on mutual information entropy to evaluate the confidence of homologous line graphs. The core idea of ​​this method is that if two nodes with the same attributes in a homologous line graph are close to the same in content, their similarity is high, and thus the confidence is also high; otherwise, the confidence is low.

[0033] Define the homologous linear subgraph to which the current node belongs The corresponding confidence value is the source confidence threshold, which is calculated by The similarity of all node pairs in the source subgraph is determined by averaging the similarities of all node pairs in the source subgraph. Untrusted nodes and their associated edges whose confidence is lower than the source confidence threshold need to be eliminated, and finally a trusted candidate subgraph is returned.

[0034] like Figure 4 As shown, the present invention designs a credible candidate subgraph matching algorithm to calculate the data source credibility of the homologous subgraph to ensure the quality of the knowledge graph embedded in the LLM. The specific implementation steps are as follows: Step 3.1. Graph the homology knowledge lines The confidence of all data nodes, data authority, and historical credibility estimation of data nodes, The corresponding source confidence threshold and the trusted candidate subgraph T-sub are initialized; Traversal , for each data node , initialize the variable N to be equal to the value corresponding to the current element The cardinality of the collection; for the current element The corresponding For each node in , execute steps 3.2 to 3.4.

[0035] Step 3.2. Calculate homologous linear subgraphs The consistency score between two data nodes in .

[0036] set up for The node set in; for any two nodes with the same attribute , defining the similarity between them A calculation method based on mutual information entropy is adopted.

[0037] Mutual Information Entropy It measures the mutual dependence of the attribute contents of two nodes, and its formula is: (1)

[0038] in, and The nodes are and The attribute value set of Is a node Get attribute value and nodes Get attribute value The joint probability distribution of and They are and The marginal probability distribution of .

[0039] Node Similarity It is defined as the normalized form of the mutual information entropy to ensure that its value is in the interval [0, 1]: (2) in and The nodes are and The entropy of the attribute value set, the node is unified using The calculation formula is: (3) in, Representative Node and , represent and .

[0040] Next, calculate the node consistency score , which reflects the consistency of nodes in different data sources, and uses mutual information entropy to calculate the similarity between node pairs. , and its consistency score is expressed as: (4) in, Is with the node A collection of nodes with the same attributes. , is the similarity between nodes.

[0041] Finally, the source confidence threshold is calculated : (5) in, represents the number of nodes in the homologous linear subgraph, Represents the similarity between node pairs.

[0042] Step 3.3. Calculate the authority score of the node.

[0043] The authority score of the node Calculated by the following formula: (6) in, is a weight coefficient used to balance the contribution of LLM evaluation authority and historical authority, satisfying ; is the node authority score evaluated by LLM, The historical authority of the node.

[0044] Based on the global influence and local connection strength of the node Evaluation; LLM calculates the credibility of knowledge by integrating the strength of association between entities, entity type information, and multi-step path information: (7) in, LLM is the node Provide authoritative ratings, is a parameter that controls the steepness of the rating curve.

[0045] The historical authority of the node It is an authoritative score based on the node's historical data. The formula is as follows: (8) in, yes The number of entities provided for all historical queries, yes The historical credibility of is the correct answer set, is the query-related data obtained in the homologous linear subgraph; Represents the entity node in the correct answer set, Indicates the authority score of the entity node.

[0046] Step 3.4. Add formula (4) and formula (6) to obtain the final confidence score of the node; if the final confidence score of the node is lower than the source credibility threshold, delete the node from the current homology subgraph.

[0047] Step 3.5. Repeat steps 3.2 to S3.4 until all elements in the data source are traversed, thereby completing the matching process of the entire trusted candidate subgraph, and finally outputting the data source credibility and the trusted candidate subgraph T-sub.

[0048] The method of the present invention screens and matches credible candidate subgraphs according to the query requirements through the above step 3, removes low-quality nodes, extracts credible candidate subgraphs from the knowledge graph, and ensures that the multi-domain retrieval query results are valid.

[0049] Step 4. Embed the core content and key related information of the selected credible candidate subgraphs into the prompt word template, and then use the vertical large model to select the data results that highly match the query intent as the data query results.

[0050] Specifically, we first design a suitable prompt word template based on the input requirements of the vertical large model and the data characteristics of the query requirements. Then, we embed the key information of the credible candidate subgraph into the prompt word template, and call the vertical large model to quickly filter out data results that highly match the query intent.

[0051] like Figure 5 As shown, the present invention integrates the first three steps and combines the vertical large model to perform accurate query of multi-source cross-domain data, including the following steps: Step 4.1. Construct a prompt word containing a small number of reference examples, perform intent recognition and entity extraction tasks based on the query requirements, and obtain a query node set; initialize the query node set and the trusted answer set.

[0052] Step 4.2. According to step 2, obtain the homologous knowledge line graph; according to step 3, calculate the confidence score of the query node and the source credibility threshold of the corresponding homologous linear subgraph, and obtain the credible candidate subgraph.

[0053] Step 4.3. Store the source confidence threshold and the credible candidate subgraph into the credible answer set; construct corresponding prompt words according to the query requirements and the credible answer set, input them into the vertical large model to ask questions, and obtain the final query answer.

[0054] The method of the present invention realizes efficient fusion of multi-source cross-domain data with the assistance of a large language model, screens and matches trusted nodes for the fused homologous knowledge line graph, obtains trusted candidate subgraphs and uses them for data retrieval queries of a vertical large model, providing an effective way for multi-source cross-domain data retrieval queries.

[0055] Example 2 This embodiment 2 describes a multi-source cross-domain data query system, which is based on the same inventive concept as the multi-source cross-domain data query system in the above embodiment 1.

[0056] The multi-source cross-domain data query system in this embodiment includes the following modules: The multi-source knowledge graph construction module is used to retrieve multi-source cross-domain data from different sources and storage forms in the field, and use the large language model to identify, extract and classify the data to build a multi-source knowledge graph; A homologous knowledge line graph construction module is used to reconstruct and aggregate the homologous knowledge line graph obtained, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph, and obtain a homologous knowledge line graph; The trusted candidate subgraph screening module is used to use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the trusted candidate subgraphs related to the query; And a data query module, which is used to embed the core content and key related information of the screened credible candidate subgraphs into the prompt word template, and then use the vertical large model to screen out data results that highly match the query intent.

[0057] It should be noted that the implementation process of the functions and effects of each functional module in the multi-source cross-domain data query system is specifically described in the implementation process of the corresponding steps in the multi-source cross-domain data query method in the above embodiment 1, and will not be repeated here.

[0058] Example 3 This embodiment 3 describes a computer device, which includes a memory and one or more processors. An executable code is stored in the memory, and when the processor executes the executable code, it is used to implement the above-mentioned multi-source cross-domain data query method. In this embodiment, the computer device is any device or apparatus with data processing capabilities, which will not be repeated here.

[0059] Example 4 This embodiment 4 describes a computer-readable storage medium, on which a program is stored. When the program is executed by a processor, it is used to implement the above-mentioned multi-source cross-domain data query method.

[0060] The computer-readable storage medium may be an internal storage unit of any device or apparatus with data processing capabilities, such as a hard disk or memory, or an external storage device of any device with data processing capabilities, such as a plug-in hard disk, a smart media card (SMC), an SD card, a flash card, etc., equipped on the device.

[0061] Of course, the above description is only a preferred embodiment of the present invention, and the present invention is not limited to the above embodiments. It should be noted that all equivalent substitutions and obvious deformation forms made by any technician familiar with the field under the guidance of this specification fall within the essential scope of this specification and should be protected by the present invention.

Claims

1. A multi-source cross-domain data query method, characterized in that: The steps include: Step 1. Retrieve multi-source cross-domain data from different sources and storage forms in the field, and use a large language model to identify, extract and classify the data to build a multi-source knowledge graph; Step 2. Reconstruct and aggregate the multi-source knowledge graph, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph, and obtain a homologous knowledge line graph; Step 3. Based on the query requirements, use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the credible candidate subgraphs related to the query; Step 4. Embed the core content and key related information of the selected credible candidate subgraphs into the prompt word template, and then use the vertical large model to select the data results that highly match the query intent as the data query results.

2. The multi-source cross-domain data query method according to claim 1, characterized in that: The step 1 is specifically as follows: First, multi-source cross-domain data is obtained from different data sources and storage media; wherein the multi-source cross-domain data types include structured tables, semi-structured XML documents and unstructured texts; Use a large language model to extract entities and relationships, and obtain entities and relationships between entities in multi-source cross-domain data; use a unified data storage method to preliminarily integrate multi-source cross-domain data to obtain a multi-source knowledge graph.

3. The multi-source cross-domain data query method according to claim 2 is characterized in that: In step 1, the process of preliminary fusion of multi-source cross-domain data is as follows: Different data conversion parsers are used to parse each different data type, namely structured data, semi-structured data and unstructured data; For structured data, its parsing is equivalent to storing table data in JSON format, and the attribute variables in the file are stored using the column storage model DSM; For semi-structured data, its parsing is equivalent to storing tree data in JSON format. The file is stored in multi-layer nested JSON format. There is no column index and fast retrieval for data in this format. Tree / graph retrieval algorithm is used for retrieval. For unstructured data, which is text data, the content is directly stored, and LLM is considered to be used for entity / relationship extraction tasks to obtain corresponding information in the future; Finally, the fusion of multi-source cross-domain data is expressed as the following formula: ; in, represents the fused multi-source knowledge graph, Represents the data conversion parser for structured data, semi-structured data and unstructured data respectively, A data set that represents structured data, semi-structured data, and unstructured data respectively.

4. The multi-source cross-domain data query method according to claim 1, characterized in that: The step 2 is specifically as follows: The entities and relations extracted by the large language model in step 1 are matched and aligned with the same or similar entities in different data sources through a text similarity model, and the relations between entities are sorted and merged to obtain an aggregated homologous data set; Construct homologous knowledge line graph based on homologous data group; Each node in the homology knowledge line graph represents a triple, and the associated edges represent the relationship between triples.

5. The multi-source cross-domain data query method according to claim 4 is characterized in that: In step 2, the construction process of the homology knowledge line graph is as follows: Step 2.

1. Initialize the unvisited node set, homologous data set, and isolated point set, including initializing the unvisited node set , homologous data set SVs , isolated point set ; Step 2.

2. Traverse each node in the unvisited node set, and for the currently traversed node A, initialize the set of nodes and edges used to store the same source nodes and related variables of the same source nodes; traverse the other nodes B in the unvisited node set except node A again, and when it is determined that node A and other nodes B meet the same source relationship, perform the following operations: Add the current node A and other nodes B that satisfy the homology relationship to the homology node set respectively and edge sets ; And remove the current node A and other nodes B from the unvisited node set to avoid repeated access and processing; Step 2.

3. After determining the homologous nodes and edge sets of a node, make the following judgments: If the same source node set If it is an empty set, add the current node A to the isolated point set In the same source data set, it is managed separately as a special data point independent of the same source data set; if the same source node set If it is not an empty set, the same source node set And the corresponding edge set As a homogenous data set , added to the homologous data set middle; Step 2.

4. Repeat steps 2.2 and 2.3 until all nodes in the unvisited node set have been traversed and processed, completing the construction process of the entire homology knowledge graph and finally outputting the homology data set. and isolated point sets ; Step 2.

5. For homogeneous data sets Each homologous data set in , which contains the set of homologous nodes and homologous edge sets Constructing homologous linear subgraphs ; All homologous linear subgraphs and isolated point sets Aggregate to get a homologous knowledge line graph .

6. The multi-source cross-domain data query method according to claim 5, characterized in that: The step 3 is specifically as follows: The query requirements are parsed into retrieval instructions, and the homologous knowledge line graph is traversed according to the confidence of the nodes to screen and match the query-related subgraphs. The obtained subgraphs are called credible candidate subgraphs. The process is as follows: The confidence of the entity node is calculated by the mutual information entropy confidence calculation method, and the reliability of the node source, node authority and historical authority are comprehensively considered. The probability statistical model and large language model are used to calculate the score; Define the homologous linear subgraph to which the current node belongs The corresponding confidence value is the source confidence threshold, which is calculated by calculating the homologous linear subgraph The average similarity of all node pairs in is determined; Untrusted nodes and their associated edges whose confidence is lower than the source confidence threshold are removed, and finally a trusted candidate subgraph is returned.

7. The multi-source cross-domain data query method according to claim 6, characterized in that: In step 3, the specific process of querying and matching the trusted candidate subgraph is as follows: Step 3.

1. Graph the homology knowledge lines The confidence of all data nodes, data authority, and historical credibility estimation of data nodes, The corresponding source confidence threshold and the trusted candidate subgraph T-sub are initialized; Traversal , for each data node , initialize the variable N to be equal to the value corresponding to the current element The cardinality of the collection; for the current element The corresponding For each node in, execute steps 3.2 to 3.4; Step 3.

2. Calculate homologous linear subgraphs The consistency score between two data nodes in ; set up for The node set in; for any two nodes with the same attribute , defining the similarity between them The calculation method based on mutual information entropy is adopted, and its formula is: (1) in, and The nodes are and The attribute value set of Is a node Get attribute value and nodes Get attribute value The joint probability distribution of and They are and The marginal probability distribution of Node Similarity It is defined as the normalized form of the mutual information entropy to ensure that its value is in the interval [0, 1]: (2) in and The nodes are and The entropy of the attribute value set, the node is unified using The calculation formula is: (3) in, Representative Node and , represent and ; Next, calculate the node consistency score , which reflects the consistency of nodes in different data sources, and uses mutual information entropy to calculate the similarity between node pairs. , and its consistency score is expressed as: (4) in, Is with the node A collection of nodes with the same attributes. , is the similarity between nodes; Finally, the source confidence threshold is calculated : (5) in, represents the number of nodes in the homologous linear subgraph, Represents the similarity of node pairs; Step 3.

3. Calculate the authority score of the node; The authority score of the node Calculated by the following formula: (6) in, is a weight coefficient used to balance the contribution of LLM evaluation authority and historical authority, satisfying ; is the node authority score evaluated by LLM, The historical authority of the node; Based on the global influence and local connection strength of the node Evaluation; LLM calculates the credibility of knowledge by integrating the strength of association between entities, entity type information, and multi-step path information: (7) in, LLM is the node Provide authoritative ratings, is a parameter that controls the steepness of the rating curve; The historical authority of the node It is an authoritative score based on the node's historical data. The formula is as follows: (8) in, yes The number of entities provided for all historical queries, yes The historical credibility of is the correct answer set, is the query-related data obtained in the homologous linear subgraph; Represents the entity node in the correct answer set, represents the authority score of the entity node; Step 3.

4. Add formula (4) and formula (6) to obtain the final confidence score of the node; if the final confidence score of the node is lower than the source confidence threshold, delete the node from the current homology subgraph; Step 3.

5. Repeat steps 3.2 to 3.4 until all elements in the data source are traversed, thereby completing the matching process of the entire trusted candidate subgraph, and finally outputting the data source credibility and the trusted candidate subgraph T-sub.

8. The multi-source cross-domain data query method according to claim 1, characterized in that: The step 4 is specifically as follows: Step 4.

1. Construct a prompt word containing reference examples, perform intent recognition and entity extraction tasks based on the query requirements, and obtain a query node set; initialize the query node set and the trusted answer set; Step 4.

2. According to step 2, a homologous knowledge line graph is obtained; and according to step 3, the confidence score of the query node and the source credibility threshold of the corresponding homologous linear subgraph are calculated, and a credible candidate subgraph is obtained; Step 4.

3. Store the source confidence threshold and the credible candidate subgraph into the credible answer set; According to the query requirements and the set of credible answers, corresponding prompt words are constructed, and the vertical large model is input to ask questions to obtain the final query answer.

9. A multi-source cross-domain data query system, characterized in that: Includes the following modules: The multi-source knowledge graph construction module is used to retrieve multi-source cross-domain data from different sources and storage forms in the field, and use the large language model to identify, extract and classify the data to build a multi-source knowledge graph; A homologous knowledge line graph construction module is used to reconstruct and aggregate the homologous knowledge line graph obtained, merge and reorganize the nodes with similar semantics scattered in the multi-source knowledge graph, and obtain a homologous knowledge line graph; The trusted candidate subgraph screening module is used to use the large language model to perform confidence scoring of query nodes in the homologous knowledge line graph, calculate the source credibility threshold of the corresponding homologous linear subgraph, and screen the trusted candidate subgraphs related to the query; And a data query module, which is used to embed the core content and key related information of the screened credible candidate subgraphs into the prompt word template, and then use the vertical large model to screen out data results that highly match the query intent.

Citation Information

Patent Citations

  • Intelligent query reasoning method and device based on knowledge graph and large language model collaboration

    CN117743531A

  • Conflict elimination query method and device for multi-source heterogeneous data

    CN118484542A

  • Multi-modal data distributed retrieval method and system based on mapping knowledge domain and vector matching

    CN118551086A

  • Query method integrating AI large model and knowledge graph

    CN118916499A

  • Intelligent equipment fault question-answering method and system based on knowledge graph enhancement

    CN119226481A

Cited By

  • Data space-oriented data retrieval enhancement generation method, device and equipment

    CN119988697A

  • Multi-source heterogeneous spatial data retrieval method and system based on R tree improvement

    CN120011370A

  • Knowledge construction method and system based on large model and RAG technology

    CN120296111A

  • Request response method and device, electronic equipment and storage medium

    CN120407755A

  • File credible question and answer method based on graph context retrieval and knowledge graph enhancement

    CN121501938A