A knowledge-driven web page table extraction and structuring processing method

Abstract

The present application relates to a kind of knowledge-driven based on web page table extraction and structured processing method, belong to the information processing technical field in computer science.This method includes: parsing Web page content data, judge whether there is table;If there is table, extract the content of all tables and store;After obtaining table, each row data and each column data are parsed, judge whether entity or attribute;Table item content is matched, and the label corresponding to each table item is obtained;According to table item label, obtain row and column label;Calculate row and column confidence, judge table header entity attribute;And generate structured triple including entity, attribute and attribute value.The present application extracts and parses web table data based on knowledge base by fusing rule and natural language processing technology, so that the information after structuring is clearer, more accurate and practical, provides technical support for the construction of various equipment performance parameters, has great practical value.

Description

TECHNICAL FIELD

[0001] The application provides a webpage table information extraction method, and belongs to the technical field of information processing in computer science, and particularly relates to a method for converting webpage tables into structured triples based on knowledge driving and natural language processing technology. BACKGROUND

[0002] Modern society develops rapidly, and science and technology are flourishing. The Internet has spread to every corner of human society. The advent of the big data era has turned the massive data of this society into huge potential wealth. There are a large amount of semi-structured and unstructured data on the Internet. Unstructured data generally needs to be learned and trained by a machine learning model to extract important information, which consumes a lot of computing power. For semi-structured data, rules can be established for analysis and processing, which is simple and easy compared to the other, and also rich information can be obtained. Among them, webpage table data is a kind of semi-structured data, which often contains great value.

[0003] In the long-term scientific research production process, it is found that the expression ability of triples is very rich, and almost all things can be expressed as triples naturally or forcibly, such as “the weather is really cold today” can be expressed as “weather-state-cold”, and triples are very suitable for subsequent knowledge graph work, such as Neo4j, a graph database that uses triples as storage units, and graph query and inference tools use triples more conveniently than ordinary relational databases.

[0004] Existing webpage table extraction methods generally represent webpage tables as DOM trees, and combine specific rules or methods to achieve table data extraction. There is no report on parsing table data into entity, attribute, and attribute value structured triples, and entity, attribute, and attribute value triples have high practical value. For example, in the military field, it can provide accurate data support for the construction of military field weapon equipment knowledge graph. SUMMARY

[0005] The application is an original invention, and the purpose is to realize webpage table data structuring by fusing rules and natural language processing technology. Specifically, by analyzing and extracting table data in a webpage, accurately identifying whether the row table header and the list header correspond to an entity or an attribute, structured triples of entity, attribute, and attribute value are obtained.

[0006] The purpose of the application is achieved by the following technical solutions:

[0007] A webpage table extraction and structured processing method based on knowledge driving includes the following steps:

[0008] Step 1) Web page analysis, using bs4 library to parse the HTML file of the relevant publication web page crawled, and judging whether there is a web table according to the table tag;

[0009] Step 2) Get the table, if there is a web table in the webpage data, save it, the specific method is: according to the table tag, locate to each table in turn, get all the contents of the th and td tags, where th represents the table header, that is, the first row of the table, and td represents the table item, that is, the table content except the first row; Store each table data as a list type, and the list is a sub list, which stores the data of each row in the table, that is, the table is stored by row; tr represents a row of the table, and the td tags in the tr tag are the contents of the row; Define the first item content of each row in the table as the row header, and the first item content of each column as the list header;

[0010] Step 3) Formulate rules, after obtaining the table, analyze each row of data and each column of data to judge whether it is an entity or an attribute. Considering that the performance parameters of various equipment are mostly in the form of numerical value plus unit, regular matching can be used to realize the judgment of the mode, the specific method is: write basic regular expressions, including all uppercase, all lowercase, integer, floating point number, percentage, international unit, month, date, year interval, expression, etc.; Generate advanced regular expressions according to the basic regular expressions, such as: number + unit, expression + unit, etc.;

[0011] Step 4) Get the item mode, use the python re module, according to the advanced regular expression obtained in step 3), do regular matching for all items in a table, store the label mode corresponding to each item according to the storage table format, and for the unmatched ones, the label is assigned as "Not_Know", the first row or column of the table is the candidate entity, and no label assignment processing is done;

[0012] Step 5) judging the row and column labels, counting the labels of each row except the first row; if "Not_Know" occupies more than half of the row ("Not_Know" label data is more than half of the number of table entries in this row), the label of this row is None, indicating that it is impossible to determine whether all entries in this row belong to a uniform pattern; otherwise, remove the "Not_Know" label in this row, if the remaining labels are consistent, it means that the data pattern of this row is the same, return True; if the remaining labels are inconsistent, it means that the data in this row does not belong to the same format, and the row header may not be an attribute, return False; count the labels of each column except the first column; if "Not_Know" occupies more than half of the column ("Not_Know" label data is more than half of the number of table entries in this column), the label of this column is None, indicating that it is impossible to determine whether all entries in this column belong to a uniform pattern; otherwise, remove the "Not_Know" label in this column, if the remaining labels are consistent, it means that the data pattern of this column is the same, return True; if the remaining labels are inconsistent, it means that the data in this column does not belong to the same format, and the list header may not be an attribute, return False;

[0013] Step 6) calculating the row and column confidence, judging the entity and attribute, according to the label statistical results of each row and each column obtained in step 5, the row and column confidence formula is:

[0014] confidence = count(True) / Total

[0015] Where count(True) refers to the number of labels in the row table that are True, Total represents the total number of rows or columns excluding the first row or the first column, if the row confidence in the table is greater than the column confidence, the row header is an attribute, and the list header is an entity; if the row confidence in the table is less than the column confidence, the row header is an entity, and the list header is an attribute; if the row confidence in the table is equal to the column confidence, the row header is an attribute by default, and the list header is an entity;

[0016] Step 7) generating structured triples, if the row header in the table is an attribute and the list header is an entity, then traverse the rows other than the first row, and for each row, the entity changes with the index, the attribute does not change, and the attribute value changes, which can generate entity, attribute, and attribute value triples; if the row header in the table is an entity and the list header is an attribute, then traverse the rows other than the first row, and for each row, the entity does not change, the attribute changes, and the attribute value changes, which can generate entity, attribute, and attribute value triples.

[0017] Beneficial effects

[0018] The present application belongs to the pioneering invention, compared with the prior art, the present application realizes the parsing extraction of the table by establishing the regular expression knowledge base and formulating the rules, further judges the entity and attribute of the row and column, extracts the accurate related information, and finally generates the entity, attribute and attribute value triplets, which lays the foundation for constructing various equipment system parameters. In particular, the present application has great significance in the construction process of a certain equipment system. BRIEF DESCRIPTION OF DRAWINGS

[0019] Figure 1 is the total flow chart of the web page table extraction and structured processing method of the present application;

[0020] Figure 2 is an example of a regular expression of the present application;

[0021] Figure 3 is a table parsed by the embodiment of the present application;

[0022] Figure 4 is the parsing result of the embodiment of the present application. DETAILED DESCRIPTION

[0023] In order to make the technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below by specific embodiments in conjunction with the drawings of the specification. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application.

[0024] See Figure 1 The present application "a web page table extraction and structured processing method based on knowledge driving" includes seven steps: web page parsing; obtaining table; formulating rules; obtaining table item mode; judging row and column labels; calculating row and column confidence, judging entity attribute; and generating structured triplets. Now it will be described in detail.

[0025] See Figure 2 , 3 , and 4, a web page table extraction and structured processing method based on knowledge driving, the specific steps are as follows:

[0026] Step 1) parse the web page, use bs4 library to parse the crawled HTML file, and judge whether there is a web table according to the table label;

[0027] Step 2) according to the table label, locate to each table in turn, get all the contents of the labels th and td, wherein th represents the table header, i.e. the first row of the table, and td represents the table item, i.e. the table content except the first row; store each table data as a list type, and the list is a sub list, which stores the data of each row in the table, i.e. store the table by row; tr represents a row of the table, and the td labels in the tr label are all the contents of the row;

[0028] Step 3) Formulate rules. Since the performance parameters of certain equipment in a certain equipment field are mostly in the form of numerical values plus units, in this embodiment, the pattern judgment is realized by regular matching. First, write the basic regular expression, including all uppercase, all lowercase, integer, floating point number, percentage, international unit, month, date, year interval, expression, etc. Then generate advanced regular expressions according to the regular expressions, such as number + unit, expression + unit, etc.

[0029] Step 4) According to the advanced regular expression, perform regular matching on all table item contents in the table through the python re module. Store the label pattern corresponding to each table item according to the format of the stored table. The results are as follows:

[0030] ['TEXT','TEXT','TEXT','TEXT','Digit','Digit']

[0031] ['TEXT','TEXT','TEXT','TEXT','Digit','Digit']

[0032] ['TEXT','Digit_TEXT','TEXT','TEXT','Digit','Digit']

[0033] ['TEXT','TEXT','TEXT','TEXT','Digit','Digit']

[0034] Step 5) According to the table item label, judge the row and column labels. The method is as follows: If the "Not_Know" label data in the row exceeds one half of the number of table items in the row, the label of the row is None. Otherwise, remove the "Not_Know" label in the row. If the remaining labels are consistent, return True. If the remaining labels are inconsistent, return False. Count the table item labels of each column except the first column. If the "Not_Know" label data in the column exceeds one half of the number of table items in the column, the label of the column is None. Otherwise, remove the "Not_Know" label in the column. If the remaining labels are consistent, return True. If the remaining labels are inconsistent, return False. The statistical results are as follows:

[0035] Row labels: [False, False, False, False]

[0036] Column labels: [True, False, True, True, True, True]

[0037] Step 6) Calculate the row and column confidence according to the row and column labels. The calculation formula is:

[0038] Confidence = confidence = count(True) / Total

[0039] Wherein, count(True) indicates the number of labels as True in the row table, Total indicates the total number of rows or total number of columns except the first row or the first column, and the calculation result is as follows:

[0040] Row confidence: 0

[0041] Column confidence: 0.83

[0042] It can be seen that the row confidence < column confidence, the row table header is an entity, and the list header is an attribute.

[0043] Step 7) generating a structured triple, specifically: if the row table header in the table is an attribute and the list header is an entity, then traversing other rows except the first row, with the change of the index, the entity changes, the attribute does not change, and the attribute value changes, that is, the entity, attribute, and attribute value triple can be generated; if the row table header in the table is an entity and the list header is an attribute, then traversing other rows except the first row, with the change of the index, the entity does not change, the attribute changes, and the attribute value changes, that is, the entity and attribute, attribute value triple can be generated, and the result is shown in the form of a dictionary as shown in Table 1. Figure 4

[0044] From the embodiments, it can be seen that the present application is very valuable when used in the field of certain equipment, and good results are achieved.

[0045] So far, the inventor has elaborated the present application in detail through the embodiments. It should be noted that the purpose of introducing details in the embodiments is not to limit the protection scope of the claims, but to help the reader understand the present application. Those skilled in the art should know that various modifications, changes or replacements of the present application without departing from the spirit and essence of the present application will fall within the protection scope of the present application.​

Claims

1. A knowledge-driven based method for web table extraction and structuring, characterized in that: The method comprises the steps of: S1, analyzing the Web page content data, judging whether there is a table; S2, if there is a table, extracting the content of all tables and storing; S3, after obtaining the table, analyzing each row of data and each column of data, judging whether it is an entity or an attribute; S4, matching the table item content to obtain the label corresponding to each table item, the method for matching the table item content is: performing regular matching on all table items in a table, storing the label mode corresponding to each table item according to the format of the stored table, and for the unmatched, the label is assigned as "Not_Know"; the first row or the first column of the table is the candidate entity or attribute by default, and no label assignment processing is performed; S5, obtaining the row and column labels according to the table item label, the method for obtaining the row and column labels is: except for the first row, the table item labels of each row of the remaining rows are counted, if "Not_Know" occupies more than half of the row, that is, the "Not_Know" label data is more than one half of the number of table items in the row, then the label of the row is None; otherwise, remove the "Not_Know" label in the row; if the remaining labels are consistent, return True; if the remaining labels are inconsistent, return False; except for the first column, the table item labels of each column of the remaining columns are counted; if "Not_Know" occupies more than half of the column, that is, the "Not_Know" label data is more than one half of the number of table items in the column, then the label of the column is None; otherwise, remove the "Not_Know" label in the column; if the remaining labels are consistent, return True; if the remaining labels are inconsistent, return False; S6, calculating the row and column confidence, judging the table header entity attribute, first calculating the row and column confidence by the formula: Confidence = , wherein, the number of tags in the row list that are True, denotes the total number of rows or columns excluding the first row or column; if the row confidence > column confidence in the table, the row header is an attribute and the column header is an entity; if the row confidence < column confidence in the table, the row header is an entity and the column header is an attribute; if the row confidence = column confidence in the table, the row header is an attribute and the column header is an entity by default; and S7, generating a structured triple.

2. The knowledge-driven based web page table extraction and structuring method according to claim 1, characterized in that: Analyzing the Web page content data means using the bs4 library to crawl the related HTML file; Judging whether there is a table means judging whether there is a web table according to the table tag.

3. The knowledge-driven based web page table extraction and structuring method of claim 1, wherein: The method for extracting the content of all tables and storing is: according to the table tag, sequentially positioning to each table, obtaining all contents with th and td labels, storing each table data as a list type, the list is a sub-list, storing the data of each row in the table, that is, storing the table by row; the td label in the tr label is the content of the row, wherein th represents the header, that is, the first row of the table, td represents the table item, that is, the table content except the first row, and tr represents a row of the table.

4. The knowledge-driven based web page table extraction and structuring method of claim 1, wherein: The method for analyzing and judging each row of data and each column of data is: writing a basic regular expression, the content includes all uppercase, all lowercase, integer, floating point number, percentage, international unit, month, date, year interval, expression, and then generating a senior regular expression according to the basic regular expression.

5. The knowledge-driven based web page table extraction and structuring method of claim 4, wherein: The method for generating structured triples is: if the row header of the table is an attribute, and the list header is an entity, then traversing the rows other than the first row, the entity changes with the change of the index, the attribute does not change, and the attribute value changes, so that the entity, attribute, and attribute value triples can be generated; if the row header of the table is an entity, and the list header is an attribute, then traversing the rows other than the first row, the entity does not change with the change of the index, the attribute changes, and the attribute value changes, so that the entity, attribute, and attribute value triples can be generated.

Images