Method and system for constructing small-scale semi-structured data knowledge graph

Abstract

The invention relates to the technical field of artificial intelligence big data, in particular to a method and system for constructing a small-scale semi-structured data knowledge graph, and the method comprises the steps: obtaining small-scale semi-structured data of a to-be-constructed knowledge graph, determining an entity object and attribute parameters, and carrying out the preprocessing and text similarity calculation of the data, the method comprises the following steps of: obtaining the similarity of data texts, constructing a corresponding regular expression, performing entity and relationship extraction by using the regular expression to obtain an extraction result, and performing graph construction of small-scale semi-structured data according to a final extraction output result. According to the method, entity and relation extraction is carried out through the regular expression, particularly extraction of the numeric attributes is carried out, compared with an existing method of directly using a model to extract and construct the knowledge graph, the method makes up for the defects of the knowledge graph in the field of small sample semi-structured data, the capacity of extracting the numeric characteristic values is enhanced, and the method is suitable for large-scale popularization and application. Therefore, the effectiveness and the accuracy of constructing the small-scale semi-structured data graph are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence big data technology, and specifically to a method and system for constructing a small-scale semi-structured data knowledge graph. Background Art

[0002] With the development of the Internet, the generation and accumulation of massive data has become the norm. These data contain rich information and knowledge, but they often exist in unstructured or semi-structured forms and are difficult to be directly used for analysis and decision-making. How to effectively organize, manage and utilize these data has become an important research topic. Knowledge graph is a technology used to represent and store structured data. It can abstract entities, attributes and relationships in data into nodes and edges in the graph, thereby forming a data network with semantic information. Knowledge graph can support a variety of application scenarios, such as search engines, intelligent question answering, recommendation systems, etc. However, the construction of existing knowledge graphs is based on large-scale data samples, and there is currently a lack of effective methods for graph construction of small-scale data. In addition, the existing knowledge extraction methods are mainly aimed at text information, and the extraction ability of numerical features is limited, resulting in defects in the field of small sample semi-structured data. These problems limit the application of knowledge graphs in specific fields, especially when the data volume is limited or the data structure is complex. In summary, there is an urgent need for a method to construct a knowledge graph of small-scale semi-structured data to improve the effectiveness and accuracy of constructing a knowledge graph of small-scale semi-structured data. Summary of the invention

[0003] In view of the above-mentioned defects and problems, the present invention provides a method and system for constructing a small-scale semi-structured data knowledge graph, so as to improve the effectiveness and accuracy of constructing a small-scale semi-structured data knowledge graph.

[0004] The solution adopted by the present invention to solve the technical problem is: a method for constructing a small-scale semi-structured data knowledge graph, comprising the following steps: S1, obtaining small-scale semi-structured data for the knowledge graph to be constructed, wherein the structured data includes multiple entity objects and their corresponding attribute parameters; S2, preprocessing the data text by cleaning and slicing to improve the text quality and segment the text; S3, determine whether the data meets the characteristics of small-scale semi-structured data; S4, construct a regular expression that meets the text characteristics; S5, setting the rules for outputting the results after extraction; S6, build a knowledge graph based on the final output results.

[0005] Furthermore, in the step S1, the source data of the knowledge graph to be constructed is first obtained, wherein the source data can be in various forms, and the source data text includes semi-structured data and unstructured data; if the source data includes unstructured data, the semi-structured data of the knowledge graph to be constructed can be obtained by processing the source data.

[0006] Furthermore, in step S2, the data text is preprocessed using the following method: S221, firstly, data cleaning is performed on the data text, one part of which is to remove stop words; another part of which is to remove punctuation marks; S22, then the cleaned data text is sliced, and the slicing principle is to divide the text according to the original ".", so as to obtain several segments of data text.

[0007] Furthermore, in step S3, the following method is used to determine whether the data meets the characteristics of small-scale semi-structured data: S31, take two segments of the data text to perform TF-IDF text similarity calculation to obtain similarity, and so on, output the similarity matrix, which is expressed by the following formula: , Among them, TF represents the ratio of the number of times a word appears in a document divided by the total number of words in the document; IDF represents the importance of a word in the corpus, that is, ; S32, visually analyzing the output similarity matrix, setting the similarity threshold of small-scale semi-structured data to 60%, and judging whether the overall similarity is greater than 60%. If so, it meets the characteristics of small-scale semi-structured data; if less than, it does not meet the characteristics of small-scale semi-structured data.

[0008] Furthermore, in step S4, a corresponding regular expression is constructed according to the specific features of the text data: , The dependent variable regex represents the constructed regular expression, re.mid represents the regular expression construction function, the independent variable a represents the string to start matching, and the independent variable b represents the string to end matching. Substituting the expression into Matching The dependent variable match represents the matching result, the independent variable regex represents the regular expression, the independent variable text represents the text to be matched, and re. search represents the matching function.

[0009] Furthermore, in step S5, the rule for outputting the results after extraction is set as follows: using regular expressions to extract entities and relationships of the text, if the regular expression rules are met, the extraction results are output; if the regular expression rules are not met, a null value is output.

[0010] A system for building a small-scale semi-structured data knowledge graph, including the following modules: The acquisition module is used to obtain small-scale semi-structured data for building the knowledge graph; Data preprocessing module, used to clean and slice data text, improve text quality and segment text; A data feature identification module is used to determine whether the data meets the characteristics of small-scale semi-structured data; Extraction rule building module, used to build regular expressions that conform to text features; Output result processing module, used to set the rules for outputting results after extraction; The graph construction module is used to construct a knowledge graph based on the final output results.

[0011] The beneficial effects of the present invention are as follows: by obtaining small-scale semi-structured data of the knowledge graph to be constructed, determining entity objects and attribute parameters, obtaining the similarity of the data text according to the preprocessing of the data and the calculation of the text similarity, and then judging whether the data conforms to the characteristics of the small-scale semi-structured data, firstly analyzing the text type to be extracted, and finding out its characteristics and rules, and then constructing corresponding regular expressions according to these characteristics and rules, and constructing corresponding extraction regular expressions according to the specific characteristics of the text data, and using the constructed regular expressions to extract entities and relationships, and then obtaining the extraction results, and finally constructing the graph of the small-scale semi-structured data according to the final extraction output results; When constructing a knowledge graph of small-scale semi-structured data, the present invention improves the ability to process semi-structured data through data cleaning, slicing and similarity calculation, making the data more suitable for the construction of the knowledge graph, and effectively identifies the characteristics of small-scale semi-structured data through TF-IDF text similarity calculation and visual analysis, providing accurate data feature support for the construction of the knowledge graph; extracting entities and relationships through regular expressions, especially enhancing the ability to extract numerical feature values, and improving the effectiveness of constructing small-scale semi-structured data graphs. Compared with the existing direct use of model extraction to construct knowledge graphs, this method makes up for the defects of knowledge graphs in the field of small sample semi-structured data, and enhances the ability to extract numerical feature values, thereby improving the effectiveness and accuracy of constructing small-scale semi-structured data graphs. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Figure 1 is a schematic diagram of the knowledge graph of the present invention; Figure 2 It is a schematic diagram of knowledge extraction based on the prior art of the present invention; Figure 3 is a flow chart of the present invention; Figure 4 It is a schematic diagram of the module structure of the present invention. DETAILED DESCRIPTION

[0013] The present invention is further described below in conjunction with the accompanying drawings and embodiments.

[0014] See also Figure 1-4 The present invention provides a method and system for constructing a small-scale semi-structured data knowledge graph: Example

[0015] Figure 1 is an example of a knowledge graph, such as Figure 1 As shown in , the knowledge graph includes: multiple different entities and the relationships between entities. The entities, relationships, and entities form a triple structure. Specifically, in Figure 1 In the figure, entity 1 has relationship 1, relationship 2, and relationship 3 with entity 2, entity 3, and entity 4, respectively; entity 5 has relationship 1, relationship 2, and relationship 3 with entity 6, and entity 7 has relationship 3 with entity 8. In the process of constructing a knowledge graph, each entity is usually analyzed and compared in detail, so that different entities are treated as different nodes, and then the triple structure of entities, relationships, and entities is constructed.

[0016] Figure 2 A schematic diagram of knowledge extraction based on the prior art is provided for this embodiment. The method includes feature extraction, model training and conditional random field constraints. Specifically, W0, W1, W2, W3, and W4 are word embeddings, and there are two labels: Person and Organization. The label format is BIO format, B (Beginning) means beginning, I (Inside) means the inside of a label, and O (Outside) means the outside. After the word vector passes through the BILSTM network, the model predicts the probability of each label, compares it, and determines the label corresponding to the maximum probability. The essence of the CRF network is Viterbi decoding, which imposes a conditional constraint on the generated label to avoid logical errors. However, the prior art uses a large model training method to extract entities and relationships, so that the data applicable to the prior art must be large-scale and the entities and relationships contained are all text types, such as Figure 2 That is to say, the existing technology is used to extract knowledge from small-scale semi-structured data, which makes the extraction method less effective and accurate.

[0017] Based on this, this embodiment provides a method for constructing a small-scale semi-structured data knowledge graph to improve the quality of constructing the knowledge graph. The method includes: S1, obtain small-scale semi-structured data for the knowledge graph to be constructed. The structured data includes multiple entity objects and their corresponding attribute parameters; In step S1, the source data of the knowledge graph to be constructed can be first obtained, where the source data can be in various forms, such as reports, textbooks, etc., that is, the source data text may include semi-structured data and may also include unstructured data; if the source data includes unstructured data, the semi-structured data of the knowledge graph to be constructed can be obtained by processing the source data. There are many specific processing methods, such as converting unstructured data into structured data through template matching, which is not limited to the embodiments of the present application.

[0018] S2, pre-process the data text by cleaning and slicing to improve the text quality and segment the text. The following methods are used to pre-process the data text: To clean and slice the data text, we must first remove stop words. First, we need to clean the data text. One part of data cleaning is to remove stop words, which are generally "this", "that", "then", etc. Another part is to remove punctuation marks. The purpose of data cleaning is to prepare for the next step of calculating the similarity of the text. Then the cleaned data text is sliced. The slicing principle is to divide the text according to the original "." to obtain several segments of data text.

[0019] S3, determine whether the data meets the characteristics of small-scale semi-structured data; Here, to determine whether the data meets the characteristics of small-scale semi-structured data, two segments of the data text are taken to perform TF-IDF text similarity calculation, and the similarity is obtained. Similarly, the similarity matrix is ​​output. The similarity matrix is ​​expressed by the following formula: , Among them, TF represents the ratio of the number of times a word appears in a file divided by the total number of words in the file, that is: ; IDF (Inverse Document Frequency) indicates the importance of a word in a corpus. IDF is the total number of documents in a corpus divided by the number of documents containing the word, and then taking the natural logarithm. It aims to evaluate the rarity of a word in a corpus. The rarer the word, the larger the IDF value, which means that the word is more important in the corpus, that is: ; The output similarity matrix is ​​visualized and analyzed, and the similarity threshold of small-scale semi-structured data is set to 60%. It is judged whether the overall similarity is greater than 60%. If it is greater than 60%, it meets the characteristics of small-scale semi-structured data; if it is less than 60%, it does not meet the characteristics of small-scale semi-structured data.

[0020] S4, construct a regular expression that meets the text characteristics; Construct corresponding extraction regular expressions based on the specific features of text data , where the dependent variable regex represents the constructed regular expression, re.mid represents the regular expression construction function, which is used to match the middle part of the text, the independent variable a represents the string that starts matching, and the independent variable b represents the string that ends matching. Substitute the expression into where the dependent variable match represents the matching result, the independent variable regex represents the regular expression, the independent variable text represents the text to be matched, and re. search represents the matching function.

[0021] S5, setting the rules for outputting the results after extraction; Use the constructed regular expression to extract entities and relations from the text. If the text meets the regular expression rules, the extraction result is output; if it does not meet the regular expression rules, a null value is output.

[0022] S6, build a knowledge graph based on the final output results.

[0023] When constructing a small-scale semi-structured data recognition graph, the entity objects and attribute parameters are determined by obtaining small-scale semi-structured data, and the similarity of the data text is obtained based on the preprocessing of the data and the text similarity calculation, and then it is determined whether the data meets the characteristics of small-scale semi-structured data. According to the specific characteristics of the text data, the corresponding extraction regular expression is constructed, and the constructed regular expression is used to extract entities and relationships, and then the extraction results are obtained. According to the final extraction output results, the graph of small-scale semi-structured data is finally constructed. When constructing a knowledge graph of small-scale semi-structured data, regular expressions are used to extract entities and relationships, especially the extraction of numerical attributes. Compared with the existing direct use of model extraction to construct knowledge graphs, this makes up for the defects of knowledge graphs in the field of small-scale semi-structured data and enhances the ability to extract numerical feature values, thereby improving the effectiveness and accuracy of constructing small-scale semi-structured data graphs. Example

[0024] This embodiment provides a technical solution for building a system for small-scale semi-structured data knowledge graph. Figure 4 As shown, including: The acquisition module is used to obtain small-scale semi-structured data for building the knowledge graph; Data preprocessing module, used to clean and slice data text, improve text quality and segment text; A data feature identification module is used to determine whether the data meets the characteristics of small-scale semi-structured data; Extraction rule building module, used to build regular expressions that conform to text features; Output result processing module, used to set the rules for outputting results after extraction; The graph construction module is used to construct a knowledge graph based on the final output results.

[0025] The data preprocessing module performs preprocessing in the following ways: data cleaning of the data text. One part of data cleaning is to remove stop words, and the other part is to remove punctuation marks. The purpose of data cleaning is to prepare for the next step of text similarity calculation. The data preprocessing module is also used to slice the cleaned data text. The slicing principle is to segment the text according to the original "." to obtain several segments of data text.

[0026] The data feature identification module is used to calculate the TF-IDF text similarity between two segments of several data texts, and then obtain the similarity, and so on, to output the similarity matrix; the data feature identification module is also used to perform visual analysis on the output similarity matrix, setting the similarity threshold of small-scale semi-structured data to 60%, and judging whether the overall similarity is greater than 60%. If so, it meets the characteristics of small-scale semi-structured data; if less than, it does not meet the characteristics of small-scale semi-structured data.

[0027] The output result processing module is used to extract entities and relations from text using regular expressions. If the regular expression rules are met, the extraction results are output; if the regular expression rules are not met, a null value is output. Example

[0028] In this embodiment, the small-scale semi-structured data to be constructed for the knowledge graph is represented by text, taking the overview of water conservancy projects in Anhui Province as an example.

[0029] By obtaining small-scale semi-structured data for the knowledge graph to be constructed, determining the entity objects and attribute parameters, and obtaining the similarity of the data text based on data preprocessing and text similarity calculation, we can then determine whether the text data meets the characteristics of small-scale semi-structured data and construct the corresponding extraction regular expression based on the specific characteristics of the text data. , where the values ​​of independent variable a and independent variable b are shown in Table 1.

[0030] Among them, the main [river|lake] indicates text that matches and contains one of the following two strings: "main rivers exist" and "main lakes exist"; where [river|lake] represents a character set that can match any one of the characters in it. In this character set, the | symbol is used to represent the logical OR relationship, so it can match either the character "river" or the character "lake".

[0031] by The partial results obtained are shown in Table 2 as follows.

[0032] It should be noted that the first row in Table 2 above is the title row, which is "Region", "Total Area", "Cultivated Land Area", "Household Registered Population", "Main Rivers and Lakes", "Lakes", "Levee Length", etc., and these titles also represent the relationships between the extracted attribute parameter entities and the region entities. Each row below the title row represents the general situation of water conservancy projects in each administrative region, mainly including attribute parameters such as the total area, cultivated land area, household registered population, main rivers and lakes, lakes, and levee length of these administrative regions. For example, taking "Changfeng County" as an example, the total planned area of Changfeng County is 1,841.4 square kilometers; the cultivated land area of Changfeng County is 110.39 hectares; the household registered population of Changfeng County is 769,600; the main rivers in Changfeng County are Dongfei River, Yao River, Zhuangmu River, Nanfei River and its tributaries Sili River, Ershibu River, etc.; the levee length of Changfeng County is 83 meters.

[0033] In this embodiment, a method for constructing a small-scale semi-structured data knowledge graph provided in this embodiment is compared with the prior art method, as shown in Table 3, where Method 1 is a method for constructing a small-scale semi-structured data knowledge graph, and Method 2 is the prior art method.

[0034]

[0035] It should be noted that in Table 3 above, the formula for precision is: , where TP: the number of true positives, that is, the number of entities correctly identified; FP: the number of false positives, that is, the number of entities that actually do not exist but are wrongly identified as existing.

[0036] The formula for recall is: , Among them, TP is the number of true positives, that is, the number of entities that are correctly identified; FN is the number of false negatives, that is, the number of entities that actually exist but are not identified.

[0037] F1-score calculation formula: , Among them, precision is the ratio of correctly identified entities to the number of identified entities; recall is the ratio of correctly identified entities to the number of entities in the sample; Error is the number of entities with errors in identification, Total1 is the number of identified entities, and Total2 is the number of entities in the sample.

[0038] From the comparison results in Table 3, it can be seen that the performance index F1 of method one is higher than that of method two (about 4%); and from the effectiveness of the two methods, since method two requires a larger sample, the amount of water conservancy project data in this example cannot meet the requirement, and it is completed by adding other annotated data, so method one is more effective than method two; from the perspective of manpower and material resources, method two requires a large amount of manually annotated data, so method two is generally more time-consuming and inefficient than method one. In summary, for this kind of data with small samples and strong rules, method one is better than method two.

[0039] It should be understood by those skilled in the art that embodiments of the present invention may provide methods or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0040] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems) and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram and the combination of processes and / or blocks in the flowchart and / or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the process in the flowchart. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0041] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0042] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0043] The above description is only a preferred embodiment of the present invention and does not limit the present invention. Any modifications, equivalent substitutions and improvements made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing a small-scale semi-structured data knowledge graph, characterized in that: The following steps are involved: S1, obtaining small-scale semi-structured data for the knowledge graph to be constructed, wherein the structured data includes multiple entity objects and their corresponding attribute parameters; S2, preprocessing the data text by cleaning and slicing to improve the text quality and segment the text; S3, determine whether the data meets the characteristics of small-scale semi-structured data; S4, construct a regular expression that meets the text characteristics; S5, setting the rules for outputting the results after extraction; S6, build a knowledge graph based on the final output results.

2. A method for constructing a small-scale semi-structured data knowledge graph according to claim 1, characterized in that: In the step S1, the source data of the knowledge graph to be constructed is first obtained, wherein the source data can be in various forms, and the source data text includes semi-structured data and unstructured data; if the source data includes unstructured data, the semi-structured data of the knowledge graph to be constructed can be obtained by processing the source data.

3. The method for constructing a small-scale semi-structured data knowledge graph according to claim 1, characterized in that: In step S2, the data text is preprocessed using the following method: S221, firstly, data cleaning is performed on the data text, one part of which is to remove stop words; another part of which is to remove punctuation marks; S22, then the cleaned data text is sliced, and the slicing principle is to segment the text according to the original ".", so as to obtain several segments of data text.

4. The method for constructing a small-scale semi-structured data knowledge graph according to claim 1, characterized in that: In step S3, the following method is used to determine whether the data meets the characteristics of small-scale semi-structured data: S31, take two segments of the data text to perform TF-IDF text similarity calculation to obtain similarity, and so on, output the similarity matrix, which is expressed by the following formula: ， Among them, TF represents the ratio of the number of times a word appears in a document divided by the total number of words in the document; IDF represents the importance of a word in the corpus, that is, ; S32, visually analyzing the output similarity matrix, setting the similarity threshold of small-scale semi-structured data to 60%, and judging whether the overall similarity is greater than 60%. If so, it meets the characteristics of small-scale semi-structured data; if less than, it does not meet the characteristics of small-scale semi-structured data.

5. The method for constructing a small-scale semi-structured data knowledge graph according to claim 1, characterized in that: In step S4, a corresponding regular expression is constructed according to the specific features of the text data: , The dependent variable regex represents the constructed regular expression, re.mid represents the regular expression construction function, the independent variable a represents the string to start matching, and the independent variable b represents the string to end matching. Substituting the expression into Matching The dependent variable match represents the matching result, the independent variable regex represents the regular expression, the independent variable text represents the text to be matched, and re. search represents the matching function.

6. A method and system for constructing a small-scale semi-structured data knowledge graph according to claim 1, characterized in that: In step S5, the rule for outputting the results after extraction is set as follows: using regular expressions to extract entities and relations of the text, and if the regular expression rules are met, the extraction results are output; If the regular expression rule is not met, a null value is output.

7. A system for constructing a small-scale semi-structured data knowledge graph, characterized in that: Includes the following modules: The acquisition module is used to obtain small-scale semi-structured data for building the knowledge graph; Data preprocessing module, used to clean and slice data text, improve text quality and segment text; A data feature identification module is used to determine whether the data meets the characteristics of small-scale semi-structured data; Extraction rule building module, used to build regular expressions that conform to text features; Output result processing module, used to set the rules for outputting results after extraction; The graph construction module is used to construct a knowledge graph based on the final output results.

Images