A domain knowledge extraction method based on inverted list

Abstract

This invention discloses a domain knowledge extraction method based on inverted indexes, comprising the following steps: 1) establishing inverted indexes for unstructured, semi-structured, and structured data in a ship knowledge base; the inverted indexes store the position mapping of words in documents; 2) retrieving candidate knowledge for the requested question from the inverted index cluster based on a round-robin load balancing algorithm; 3) ranking the retrieved candidate knowledge; 4) extracting triples from the ranked Top N candidate knowledge; 5) correcting and fusing conflicting triples in the extracted triples; 6) obtaining a knowledge graph after knowledge extraction based on the processed triples. This invention improves system response speed by narrowing the range of candidate knowledge and quickly extracting knowledge from the knowledge base.

Description

Technical Field

[0001] This invention relates to natural language processing technology, and more particularly to a domain knowledge extraction method based on inverted indexes. Background Technology

[0002] Currently, the challenge in refining knowledge in natural language processing lies in processing the data from the knowledge source. Knowledge doesn't exist in a ready-made form; only after analyzing, identifying, understanding, and associating the data can useful knowledge be extracted. This data processing often presents varying levels of difficulty depending on the form of the data in the knowledge source. For some knowledge embedded in databases or text data, although it has an explicit data representation, current machine learning techniques are insufficient for automatic filtering, or the automatically filtered knowledge contains many errors, resulting in poor quality of the filtered knowledge. Current knowledge refining primarily relies on manual or machine-assisted manual acquisition. Therefore, there is an urgent need to design a simple and effective method for refining knowledge in knowledge bases to overcome the shortcomings of existing technologies. Summary of the Invention

[0003] The technical problem to be solved by the present invention is to provide a domain knowledge extraction method based on inverted index, which addresses the shortcomings of the prior art.

[0004] The technical solution adopted by this invention to solve its technical problem is: a domain knowledge extraction method based on an inverted index, comprising the following steps:

[0005] 1) Create an inverted index table for the unstructured, semi-structured, and structured data in the ship knowledge base; the inverted index table stores the position mapping of words in the document;

[0006] 2) A round-robin-based load balancing algorithm retrieves candidate knowledge of request issues from the inverted index table cluster;

[0007] 3) Sort the retrieved candidate knowledge;

[0008] For the retrieved candidate knowledge, calculate the similarity score (FuzzScore) and the word ratio (WordRatio) between the question and the segmented candidate answer;

[0009] The retrieved candidate knowledge is sorted from largest to smallest according to the WordRatio score. If the WordRatio values ​​are the same, they are sorted according to their corresponding FuzzScore scores. The top N candidate knowledge is selected and the bottom candidate knowledge is removed.

[0010] 4) Use a knowledge extraction algorithm to extract triples from the sorted TopN candidate knowledge;

[0011] The knowledge extraction algorithm uses a pipeline pattern for triple extraction. It uses a sequence labeling model to predict entities in a sentence, and then uses a relation classification model to determine the category of entity relations, filtering out cases where the relations are unknown, and extracting valid triples.

[0012] The sequence labeling model employs an ALBERT+Bi-LSTM+CRF network and uses the BIO annotation system to treat the subject and object in the sentence as entities, labeled as SUBJ and OBJ respectively. The relation classification model uses an ALBERT+Bi-GRU+ATT network to determine the relation between the subject and object in the sentence based on the output of the sequence labeling model. Finally, based on the extracted subject, object, and relation, cases with unknown relations are filtered out, forming triples.

[0013] 5) Correct and integrate conflicting triples in the extracted triples;

[0014] The Path Constraint Resource Allocation (PCRA) algorithm is used to merge conflicting triples. The amount of resources that the head entity eventually receives through the path to the tail entity is used as the confidence level, and unreliable triples are filtered out based on the confidence level.

[0015] 6) Obtain the knowledge graph after knowledge extraction based on the processed triples.

[0016] The beneficial effects of this invention are:

[0017] 1) The method of the present invention can refine knowledge from structured data, semi-structured data and unstructured data, and improve the system response speed by narrowing down the range of candidate knowledge and quickly extracting knowledge from the knowledge base;

[0018] 2) Improve the quality of knowledge graphs by correcting errors and integrating knowledge from conflicting triples. Attached Figure Description

[0019] The present invention will be further described below with reference to the accompanying drawings and embodiments. In the accompanying drawings:

[0020] Figure 1 This is a flowchart of a method according to an embodiment of the present invention;

[0021] Figure 2 This is a schematic representation of the inverted index in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0023] like Figure 1 As shown, a domain knowledge ranking method based on an inverted index includes the following steps:

[0024] 1) Create an inverted index table for the unstructured, semi-structured, and structured data in the ship knowledge base; the inverted index table stores the position mapping of words in the document;

[0025] Build an inverted index table for all data, as shown in the example. Figure 2 By extracting words from documents, an index is built that links words to documents. For example, for the three documents in the example, a list of words in the documents and a list of documents containing those words are created.

[0026] 2) A round-robin-based load balancing algorithm retrieves candidate knowledge of request issues from the inverted index table cluster;

[0027] 3) Sort the retrieved candidate knowledge;

[0028] For the retrieved candidate knowledge, calculate the FuzzScore and WordRatio scores;

[0029] The retrieved candidate knowledge is sorted from largest to smallest according to the WordRatio score. If the WordRatio values ​​are the same, they are sorted according to their corresponding FuzzScore scores. The top N candidate knowledge is selected and the bottom candidate knowledge is removed.

[0030] The specific calculation process of the WordRatio score is as follows. Assume the length of question Q is L1, the number of segments after Q segmentation is S, the length of candidate answer A is L2, and the number of strings after A segmentation is N. The time complexity of the WordRatio algorithm is O(L1*L2+S*N).

[0031] Algorithm 1, WordRatio value calculation

[0032] Input: Question Q after semantic analysis and processing, and candidate answer A;

[0033] Output: WordRatio value;

[0034]

[0035]

[0036] The FuzzScore calculation method is shown in Algorithm 2. Assume the length of candidate answer A is L1, the number of strings after A is split is N, and the length of question Q is L2. The similarity score between the question and the split candidate answers is calculated based on the edit distance algorithm, with a time complexity of O(n log n). The time complexity of Algorithm 2 is O(L2*L1).

[0037] Algorithm 2: FuzzScore Calculation

[0038] Input: Question Q after semantic analysis and processing, and candidate answer A.

[0039] Output: FuzzScore value

[0040]

[0041]

[0042] 4) Use a knowledge extraction algorithm to extract triples from the sorted TopN candidate knowledge;

[0043] The knowledge extraction algorithm uses a pipeline pattern for triple extraction. It uses a sequence labeling model to predict entities in a sentence, and then uses a relation classification model to determine the category of entity relations, filtering out cases where the relations are unknown, and extracting valid triples.

[0044] The sequence labeling model employs an ALBERT+Bi-LSTM+CRF network and uses the BIO annotation system to treat the subject and object in the sentence as entities, labeled as SUBJ and OBJ respectively. The relation classification model uses an ALBERT+Bi-GRU+ATT network to determine the relation between the subject and object in the sentence based on the output of the sequence labeling model. Finally, based on the extracted subject, object, and relation, cases with unknown relations are filtered out, forming triples.

[0045] 5) Correct and integrate knowledge from conflicting triples;

[0046] The Path Constraint Resource Allocation (PCRA) algorithm is used to merge conflicting triples. The amount of resources that the head entity eventually receives through the path to the tail entity is used as the confidence level, and unreliable triples are filtered out based on the confidence level.

[0047] It should be understood that those skilled in the art can make improvements or modifications based on the above description, and all such improvements and modifications should fall within the protection scope of the appended claims.

Claims

1. A domain knowledge extraction method based on inverted indexes, characterized in that, Includes the following steps: 1) Establish an inverted index table for unstructured, semi-structured, and structured data in the ship knowledge base; The inverted index table stores the position mapping of words in a document; 2) A round-robin-based load balancing algorithm retrieves candidate knowledge of request issues from the inverted index table cluster; 3) Sort the retrieved candidate knowledge. For the retrieved candidate knowledge, calculate the similarity score (FuzzScore) and the word ratio (WordRatio) between the question and the segmented candidate knowledge answer; The retrieved candidate knowledge is sorted from largest to smallest according to the WordRatio score. If the WordRatio values ​​are the same, they are sorted according to their corresponding FuzzScore scores. The top N candidate knowledge is selected and the bottom candidate knowledge is removed. In this scenario, assuming the length of question Q is L1, the number of segments after Q segmentation is S, the length of candidate answer A is L2, and the number of strings after candidate answer A is N, the calculation of the WordRatio score involves five steps: segmentation, string splitting, longest continuous substring counting, existing word counting, and final value calculation. The steps are as follows: 31) Perform word segmentation on the input question Q to obtain the word segmentation set W={w1,w2,…,w m }, where m is the total number of word segments, i.e., S as defined above; 32) Perform text segmentation on candidate answer A, using punctuation marks as the segmentation criterion, to obtain the string set C = {c1, c2, ..., c...}. n }, where n is the number of strings after segmentation, i.e. N as defined above, to achieve segmentation of the answer text; 33) Obtain Length of the longest contiguous substring on the split statement in candidate answer A ; Initialize the length ML of the longest and longest consecutive substring to 0, and iterate through each string in the string set C: 3.31) Calculate the problem Q in the current string c. i The length L of the longest continuous substring on; 3.32) If the currently calculated L is greater than the current ML value, then update ML to L; 3.33) After the traversal is complete, ML will retain the length of the longest continuous substring that question Q can match in each segment of candidate answer A; 34) Obtain the number of word segments contained in candidate answer A. Initialize the number of existing words EW to 0, and iterate through each word in the word segmentation set W of problem Q: 3.41) Statistically, candidate answer A contains the current word segment w. i The number of items is COUNT; 3.42) If COUNT is not 0, then increment the EW value by 1; 3.43) After the traversal is complete, EW will record the total number of matching word segments in question Q that can be found in candidate answer A; 3.5) Based on the ML and EW values ​​obtained in the above steps, calculate the WordRatio score; 4) Extract triples from the sorted TopN candidate knowledge; 5) Correct and integrate conflicting triples in the extracted triples; 6) Obtain the knowledge graph after knowledge extraction based on the processed triples.

2. The domain knowledge extraction method based on inverted indexes according to claim 1, characterized in that, Step 4) uses a knowledge extraction algorithm to extract triples from the sorted TopN candidate knowledge. The knowledge extraction algorithm uses a pipeline pattern triple extraction, uses a sequence labeling model to predict entities in a sentence, and then uses a relation classification model to determine the category of entity relations, filtering out cases where the relations are unknown, and extracting valid triples. The sequence labeling model uses an ALBERT+Bi-LSTM+CRF network and a BIO labeling system to label the subject and object in the sentence as entities, labeled as SUBJ and OBJ respectively. The relation classification model uses an ALBERT+Bi-GRU+ATT network. Based on the output of the sequence labeling model, the relation between the subject and object in the sentence is determined. Finally, based on the extracted subject, object, and relation, cases with unknown relations are filtered out to form triples.

3. The domain knowledge extraction method based on inverted indexes according to claim 1, characterized in that, In step 5), a path-constrained resource allocation algorithm is used to merge conflicting triples. The amount of resources that the head entity eventually flows to the tail entity through the path is used as the confidence level, and unreliable triples are filtered out by the confidence level.

Images