Attribute information disambiguation method and system for html document

Abstract

The application belongs to the technical field of network information processing, and specifically provides a kind of html document attribute information disambiguation method and system, wherein the method comprises: html document is parsed into text data and table data and is stored by XPath;Attribute key and information value are obtained by using rule extraction or model extraction for line-by-line processing;Attribute key in text data and table data is disambiguated using context information respectively.The scheme converts html document into chapter and table, and then disambiguates text and table using some context information, improving the accuracy of information extraction.

Description

Technical Field

[0001] This invention relates to the field of network information processing technology, and more specifically, to a method and system for disambiguating attribute information in HTML documents. Background Technology

[0002] Because HTML documents are written by different people, the terminology used for certain concepts may differ. In some specific sections, the meaning of the attribute name "key" may shift, thus requiring ambiguity resolution. For example, in the "Successful Supplier Information" section, "Transaction Amount" as a key refers to the monetary value paid by the supplier to win the bid. However, in the "Successful Bid Information" section, "Transaction Amount" as a key refers to the monetary value paid by the supplier for a particular bid, i.e., the bid's monetary value. Treating them interchangeably will lead to information discrepancies.

[0003] Therefore, how to disambiguate text and tables in HTML documents is an urgent problem to be solved. Summary of the Invention

[0004] This invention addresses the technical problem of low accuracy in extracting text and table information from HTML documents in existing technologies.

[0005] This invention provides a method for disambiguating attribute information in an HTML document, comprising the following steps:

[0006] S1, use XPath to parse the HTML document into text data and table data and store them;

[0007] S2, use rule extraction or model extraction to process row by row to obtain attribute key and information value;

[0008] S3 uses context information to disambiguate the attribute keys in both text and tabular data.

[0009] Preferably, S1 specifically includes:

[0010] Text data parsing involves organizing the text data in the HTML document, i.e., the data outside the table tags, according to the principle of one line per tag, while preserving the line break information, and also performing the same processing on the information between tags.

[0011] Parse the table data to obtain the number of rows and columns of the table labels, as well as the coordinates and content of each cell.

[0012] Preferably, the disambiguation of attribute keys in the text data in S3 specifically includes:

[0013] Step 1: Initialize the variables recorded in the context:

[0014] Step 2: Read the text data from the first line to the last line in a loop.

[0015] Preferably, step two specifically includes the following steps:

[0016] (1) If chapter information exists, extract and record it in the context variable;

[0017] (2) If attribute information exists, extract it; if it is an attribute that can help eliminate ambiguity, record it in the context variable.

[0018] (3) If the attributes extracted in step (2) are ambiguous, disambiguation is performed based on the information in the context to obtain the final attributes;

[0019] (4) Record the final attributes and their corresponding values.

[0020] Preferably, the disambiguation of attribute key in the table data in S3 specifically includes the following steps:

[0021] S31, use XPath to parse the table to form an internal data structure and record its chapter information in the data structure;

[0022] S32, iterate through each element in the header record. If chapter information exists, disambiguation is performed based on the chapter information, and the attribute is marked as "OK". If chapter information does not exist or disambiguation cannot be performed based on the chapter information, the ambiguous attribute is marked with a special "Pending Determination" mark.

[0023] Preferably, the step S32 is followed by the following step:

[0024] (a) Obtain the set of possible attributes that are "to be determined";

[0025] (b) Calculate the probability that each possible attribute in step (a) and the other "determined" attribute set will occur simultaneously;

[0026] (c) Select the attribute with the highest probability as the final attribute of the current "to be determined" attribute.

[0027] Preferably, after step S3, the method further includes: adding an information verification process for each type of information that needs to be identified, and filtering out non-compliant parts of the information.

[0028] This invention also provides an attribute information disambiguation system for HTML documents, the system being used to implement an attribute information disambiguation method for HTML documents, comprising:

[0029] The parsing module is used to parse HTML documents into text and table data using XPath and then store them.

[0030] The extraction module is used to process row by row using rule extraction or model extraction to obtain attribute keys and information values;

[0031] The disambiguation module is used to disambiguate attribute keys in text data and tabular data respectively.

[0032] The present invention also provides an electronic device, including a memory and a processor, wherein the processor is used to implement the steps of a method for disambiguating attribute information of an HTML document when executing a computer management program stored in the memory.

[0033] The present invention also provides a computer-readable storage medium storing a computer management program thereon, wherein the computer management program, when executed by a processor, implements the steps of a method for disambiguating attribute information of an HTML document.

[0034] Beneficial Effects: This invention provides a method and system for disambiguating attribute information in HTML documents. The method includes: parsing the HTML document into text data and table data using XPath and storing them; performing line-by-line processing using rule extraction or model extraction to obtain attribute keys and information values; and performing disambiguation processing on the attribute keys in the text data and table data respectively. This solution disambiguates text and tables based on some contextual information, improving the accuracy of information extraction. Attached Figure Description

[0035] Figure 1 The HTML document parsing process provided by this invention;

[0036] Figure 2 A schematic diagram of the hardware structure of a possible electronic device provided by the present invention;

[0037] Figure 3 A schematic diagram of the hardware structure of a possible computer-readable storage medium provided by the present invention;

[0038] Figure 4 Information between tags provided by this invention;

[0039] Figure 5 The flowchart for rule extraction of text data provided by this invention;

[0040] Figure 6 A flowchart for extracting tabular data provided by this invention. Detailed Implementation

[0041] The specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples. The following examples are for illustrative purposes only and are not intended to limit the scope of the invention.

[0042] like Figure 1 The present invention provides a method for disambiguating attribute information in an HTML document, comprising the following steps:

[0043] S1 uses XPath to parse the HTML document into text data and table data and then stores them.

[0044] S2, use rule extraction or model extraction to process row by row to obtain attribute key and information value;

[0045] S3 uses context information to disambiguate the attribute keys in both text and tabular data.

[0046] Among them, text data parsing organizes the text data in the HTML document, that is, the data under the non-table tags, according to the principle of one line per tag, and retains the line break information. At the same time, the information between tags is also processed in the same way.

[0047] Parse the table data to obtain the number of rows and columns of the table labels, as well as the coordinates and content of each cell.

[0048] In a specific implementation scenario, the methods for disambiguating attribute information in an HTML document include:

[0049] Step 1) Obtain the HTML document and output the article type data and table type data.

[0050] The HTML document is parsed using XPath and stored in two types of data: horizontal and vertical tables. The HTML document parsing process is as follows: Figure 1 As shown.

[0051] 1.1) XPath parsing

[0052] A) Text parsing, also known as text data parsing:

[0053] Text data, i.e. data not under table tags (such as p tags, span tags, li tags, etc.), is organized according to the principle of one tag per line, and the line break information is preserved. The same processing is also done on the information between tags. The purpose of this is to ensure that more comprehensive information is obtained and to ensure the accuracy of downstream work.

[0054] like Figure 4As shown, if the information "1. Purchaser Information" between the labels is ignored, the "Name" in the following text will be ambiguous, and it will be impossible to determine whether it is the name of the tendering party or the purchaser, so the "Purchaser Name" cannot be accurately stated.

[0055] B) Table Data Parsing

[0056] The goal of table data parsing is to parse the table tag to obtain the number of rows, columns, and the coordinates and content of each cell.

[0057] HTML documents offer a high degree of flexibility in their elements, especially tables, with features such as row merging, column merging, cell merging, nested tables, horizontal tables, and vertical tables. While these operations enhance the browser user's reading experience, they also drastically increase the difficulty of parsing HTML documents.

[0058] For merge operations, a copy strategy is used during parsing. The length and width of the current cell are obtained based on the rowspan and clspan attribute values, and then it is split into cells with a length and width of 1 unit. Each cell is filled with the content copied from the cell before splitting.

[0059] For nested tables, treat each table as an independent table.

[0060] For horizontal and vertical tables, horizontal tables display attributes and information mixed in one row, such as a resume, while vertical tables treat one row of information as a complete record. Obviously, the information extraction processes of these two types of tables are different. Therefore, it is necessary to classify them. The main basis for the classification logic is the arrangement of attributes in the first row (continuous or separated), the number of columns in the first row, and other information.

[0061] In addition, single-column and double-column tables are analyzed as chapters because of their simple structure.

[0062] Step 2) Feature Extraction

[0063] The feature extraction module consists of rule extraction and model extraction.

[0064] 2.1) Rule Extraction

[0065] Rule extraction primarily relies on summarizing experience. By observing large amounts of data, common sentence structures for attribute keys and information values ​​are summarized, and some common features are retained to enhance its generalization ability. The difficulty of rule extraction lies in maintaining an effective and conflict-free rule base, which generally requires a significant amount of manpower.

[0066] Implementing rules is simpler and more efficient than maintaining a rule base, and the results and interpretability are unmatched by other methods. For textual data, rules are used for line-by-line processing, such as... Figure 5 As shown.

[0067] The entire rule processing takes the entire text as input and then iteratively processes each line. The entire process is explained below:

[0068] a. Perform rule matching on a row of data.

[0069] -If the match is successful, proceed to step b;

[0070] - If the match fails, proceed to the next line;

[0071] b. Extract the key and value according to the matching rules.

[0072] - For key processing, proceed to step c;

[0073] - For value processing, proceed to step f;

[0074] c. Normalize the key and return the value.

[0075] - Valid, proceed to step d;

[0076] - Invalid, skip to the next line;

[0077] Is there any ambiguity in d.key?

[0078] - Yes, proceed to step e;

[0079] -No, proceed to step g;

[0080] Can e.key disambiguate?

[0081] - Yes, disambiguate and proceed to step g;

[0082] -No, skip to the next line;

[0083] f. Perform necessary data cleaning on the value;

[0084] g. Record the key and value;

[0085] For processing tabular data, such as Figure 6 As shown.

[0086] For horizontal tables, leveraging the alternating nature of attributes and information, key identification and value extraction are performed cyclically. For vertical tables, each row is treated as a complete record. First, the table header is identified, meaning all keys are identified. Then, for each row, the corresponding value is extracted based on the key. Simultaneously, key normalization and disambiguation are performed, and values ​​are cleaned as needed. Ultimately, more accurate key-value information is obtained.

[0087] The disambiguation process is divided into two parts: disambiguation of text data attributes and disambiguation of table data attributes.

[0088] A) The main steps for disambiguating attribute keys in text data are as follows:

[0089] Step 1: Initialize the variables recorded in the context;

[0090] Step 2: Loop through the text data from the first line to the last line:

[0091] (1) If chapter information exists, extract and record it in the context variable, for example: “IV. Target Information”, from which the current chapter can be extracted as “Target Information”.

[0092] (2) If attribute information exists, extract it. If it is an attribute that can help eliminate ambiguity, record it in the context variable. For example, “Purchasing unit name: Municipal Central Hospital” can extract the “Purchasing unit name” attribute. “Purchasing unit name” can help eliminate the ambiguity of the upcoming “Contact person” and “Contact information”. That is, the “Contact person” that appears under “Purchasing unit name” is “Purchasing unit contact person”, and the same is true for “Contact information”.

[0093] (3) If the attributes extracted in step (2) are ambiguous, disambiguation is performed based on the information in the context to obtain the final attributes.

[0094] (4) Match the attributes with their values ​​and record them.

[0095] B) The main steps for disambiguating the attribute key in table data are as follows:

[0096] S31. Parse the table using XPath to form its internal data structure and record its section information in the data structure, for example, "table_object.section = 'object information'".

[0097] S32, iterate through each element in the header record:

[0098] (1) If there is no ambiguity, skip it;

[0099] (2) If chapter information exists, mark it as "OK" attribute, and then disambiguation is performed based on the chapter information. For example, under "Subject Information", "Name" is "Subject Name";

[0100] (3) If chapter information does not exist or cannot be disambiguated based on chapter information, the ambiguous attribute is marked with a special mark, which is "to be determined". For example, "name" is ambiguous and chapter information does not exist, so the ambiguity cannot be eliminated, and it is marked with a special mark, such as "name to be determined";

[0101] S33, Following step S32, a list of attributes is obtained. This list may contain attributes that are "to be determined." These attributes can be used to resolve ambiguities based on other attributes. Therefore, each "to be determined" attribute is iterated over as follows:

[0102] (a) Obtain the possible set of attributes for the "to be determined" attribute. For example, "name" could be "name of the subject matter", "name of the winning company", "name of the procuring entity", etc.

[0103] (b) Calculate the probability that each possible attribute in step (a) and the other "determined" attribute set will occur simultaneously;

[0104] (c) Select the attribute with the highest probability as the final attribute of the current "to be determined" attribute. For example, when "quantity", "unit price", "total price", "brand" and "name" appear together, "name" is likely to refer to "name of the object". When "address", "contact person", "candidate company ranking" and "name" appear together, "name" is likely to refer to "company name".

[0105] 2.2) Model Extraction

[0106] More generally, for large, unstructured segments of text, a model should be trained for extraction, which falls under the category of sequence labeling. For information such as names of people, places, and companies, existing open-source models can be used directly, while some special entities are trained and predicted using labeled data.

[0107] Step 3) Information Verification

[0108] To increase the accuracy of information extraction, an information verification process is added for each type of information that needs to be identified. The purpose of this process is to determine the validity of the information and filter out non-compliant parts of the information.

[0109] This method converts HTML documents into chapters and tables, and uses contextual information for attribute disambiguation, which are difficult to achieve through templates or models, thereby improving extraction accuracy.

[0110] This invention also provides an attribute information disambiguation system for HTML documents. The system is used to implement the attribute information disambiguation method for HTML documents as described above, including:

[0111] The parsing module is used to parse HTML documents into text and table data using XPath and then store them.

[0112] The extraction module is used to process row by row using rule extraction or model extraction to obtain attribute keys and information values;

[0113] The disambiguation module is used to disambiguate attribute keys in text data and tabular data respectively.

[0114] Please see Figure 2 A schematic diagram illustrating an embodiment of the electronic device provided in this invention. For example... Figure 2 As shown, an embodiment of the present invention provides an electronic device, including a memory 1310, a processor 1320, and a computer program 1311 stored on the memory 1310 and executable on the processor 1320. When the processor 1320 executes the computer program 1311, it performs the following steps: S1, parsing an HTML document into text data and table data using XPath and storing them.

[0115] S2, use rule extraction or model extraction to process row by row to obtain attribute key and information value;

[0116] S3 uses context information to disambiguate the attribute keys in both text and tabular data.

[0117] Please see Figure 3 This is a schematic diagram illustrating an embodiment of a computer-readable storage medium provided by the present invention. (See diagram below.) Figure 3 As shown, this embodiment provides a computer-readable storage medium 1400, on which a computer program 1411 is stored. When the computer program 1411 is executed by a processor, it performs the following steps: S1, parsing an HTML document into text data and table data using XPath and storing them;

[0118] S2, use rule extraction or model extraction to process row by row to obtain attribute key and information value;

[0119] S3 uses context information to disambiguate the attribute keys in both text and tabular data.

[0120] It should be noted that the descriptions of each embodiment in the above embodiments have different focuses. For parts that are not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0121] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0122] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0123] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0124] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0125] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention.

[0126] Obviously, those skilled in the art can make various modifications and variations to this invention without departing from its spirit and scope. Therefore, if these modifications and variations fall within the scope of the claims of this invention and their equivalents, this invention also intends to include these modifications and variations.

Claims

1. A method for disambiguating attribute information in an HTML document, characterized in that, Includes the following steps: S1, using XPath to parse the HTML document into text data and table data and store them, including: text data parsing, organizing the text data in the HTML document, that is, the data under the non-table tags, according to the principle of one line per tag, and preserving the line break information, and also doing the same processing on the information between tags; Parse the table data to obtain the number of rows and columns for the table labels, as well as the coordinates and content of each cell; S2, use rule extraction or model extraction to process row by row to obtain attribute key and information value; S3, using context information to disambiguate the attribute keys in the text data and the table data respectively; the disambiguation of the attribute keys in the text data specifically includes: step one, initializing the variables of the context record; step two, looping through the first line of the text data until the last line of the text. Step two specifically includes the following steps: (1) If chapter information exists, extract and record it in the context variable; (2) If attribute information exists, extract it, and if it is an attribute that can help eliminate ambiguity, record it in the context variable; (3) If the attribute extracted in step (2) is ambiguous, disambiguate it according to the information in the context to obtain the final attribute; (4) Record the final attribute and its corresponding value. The disambiguation of attribute key in the table data in S3 specifically includes the following steps: S31, parsing the table using XPath to form an internal data structure and recording its chapter information in the data structure; S32, traversing each element recorded in the table header, if chapter information exists, disambiguation is performed based on the chapter information and the attribute is marked as "OK"; if chapter information does not exist or disambiguation cannot be performed based on the chapter information, the ambiguous attribute is marked with a special "to be determined" mark. The steps following S32 are: (a) obtaining a set of possible attributes to be determined; (b) calculating the probability that each possible attribute in step (a) and other sets of determined attributes will appear simultaneously; and (c) selecting the attribute with the highest probability as the final attribute of the current attribute to be determined.

2. The method for disambiguating attribute information in an HTML document according to claim 1, characterized in that, The process following S3 also includes: adding an information verification process for each type of information that needs to be identified, and filtering out non-compliant parts of the information.

3. A disambiguation system for attribute information in an HTML document, characterized in that, The system is used to implement the attribute information disambiguation method for HTML documents as described in any one of claims 1-2, including: The parsing module is used to parse HTML documents into text and table data using XPath and then store them. The extraction module is used to process row by row using rule extraction or model extraction to obtain attribute keys and information values; The disambiguation module is used to disambiguate attribute keys in text data and tabular data respectively.

4. An electronic device, characterized in that, The system includes a memory and a processor, wherein the processor is used to execute a computer program stored in the memory to implement the steps of the attribute information disambiguation method for an HTML document as described in any one of claims 1-2.

5. A computer-readable storage medium, characterized in that, It stores a computer program that, when executed by a processor, implements the steps of the attribute information disambiguation method for an HTML document as described in any one of claims 1-2.

Images