Web page information extraction method and system

Abstract

The present invention relates to the field of computer data processing technology, and discloses a method and system for extracting web page information. The method first obtains an HTML text from which the web page information to be extracted is analyzed and encoded based on the three dimensions of position information, structural information, and content information, thereby constructing a set of node encoding vectors for the HTML text. Based on the set of node encoding vectors, a mutual information similarity calculation method is then employed to calculate the degree of node similarity through structural information encoding, and the web page type of the HTML text is determined. If the web page type is a list web page, the first key information in the HTML text is extracted; if the web page type is a text web page, the second key information in the HTML text is extracted. The present invention can improve the accuracy and robustness of web page information extraction.

Description

[0001] This application is a divisional application with application number CN202411347462.0, application date 2024 / 09 / 26, and invention name is Web page information extraction method and system based on multi-dimensional feature analysis and modeling. Technical Field

[0002] The present invention relates to the technical field of computer data processing, and in particular to a web page information extraction method and system. Background Art

[0003] In today's information age, web pages serve as the primary medium for knowledge sharing and information dissemination. Automated content extraction is crucial for data analysis, information retrieval, market research, and other fields. With the rapid development of the internet, the number of web pages has increased dramatically, and their content has become increasingly rich and diverse. This has made effective and efficient web information extraction techniques crucial. However, traditional information extraction methods have significant limitations when dealing with complex and diverse web content, limiting the breadth and depth of information acquisition.

[0004] Currently, traditional web information extraction techniques, which primarily rely on keyword matching and regular expressions, struggle to adapt to the ever-changing nature of webpages and are insufficiently capable of handling dynamically loaded content. Furthermore, existing techniques are inadequate in filtering out irrelevant noise and interfering information. Therefore, there is an urgent need to develop new web information extraction techniques to overcome these challenges. Summary of the Invention

[0005] In order to solve the technical problems existing in the prior art, the present invention provides a web page information extraction method and system, which can adapt to different types of web page structures and content changes more accurately and efficiently.

[0006] To achieve the above object, the present invention provides the following technical solutions:

[0007] The present invention discloses a method for extracting web page information, comprising:

[0008] Obtain the HTML text of the web page information to be extracted, perform feature analysis and encoding on each node of the HTML text from three dimensions: position information, structure information, and content information, and thus construct a node encoding vector set of the HTML text;

[0009] Determining the webpage type of the HTML text according to the node coding vector set, extracting first key information from the HTML text when the webpage type is a list webpage, and extracting second key information from the HTML text when the webpage type is a text webpage;

[0010] Among them, the node feature vector set of HTML text is represented as X Node, X Node ={X1,X2,…,X i ,…,X N}; where X i Represents the feature vector of the i-th node, i∈[1,N], N is the total number of nodes in the HTML text; Indicates the location information encoding of the i-th node; Represents the structural information encoding of the i-th node; Represents the content information encoding of the i-th node;

[0011] The construction method of node location information encoding is as follows:

[0012] Get the XPath path of the current node. Encode the label part of the XPath path using enumeration mapping, and directly encode the index part of the XPath path using the index number, thereby obtaining the location information encoding of the current node based on the XPath path.

[0013] The construction method of the node structure information encoding is as follows:

[0014] Get all child nodes of the current node, subtract the location information code of the current node based on the XPath path from the location information code of each child node based on the XPath path, and aggregate the subtraction results to form the structure information code of the current node;

[0015] The node content information encoding construction method is as follows:

[0016] Count the total number of characters q in the current node A and the total number of child nodes q B , analyze the number of hyperlinks q1, the number of Chinese characters q2, and the number of punctuation marks q3 in the current node, and calculate the hyperlink density ρ1, Chinese character density ρ2, and punctuation mark density ρ3 of the current node. A ,q B , q1, q2, q3, ρ1, ρ2, ρ3 construct the content information encoding of the current node; where ρ1=q1 / q B ;ρ2=q2 / q A ;ρ3=q3 / q A ;

[0017] The determining the webpage type of the HTML text according to the node coding vector set includes:

[0018] The similarity between nodes is calculated through mutual information, so as to calculate the overall node similarity MI of HTML text. The calculation formula is:

[0019]

[0020] Where Z i' and Z j' They represent two vectors in the HTML root node structure information encoding; I(·) represents the similarity calculation function between the two vectors; z1 represents the vector Z i' An element in , z2 represents the vector Z j' An element in ; p(z1) represents the marginal probability distribution function of z1; p(z2) represents the marginal probability distribution function of z2; p(z1,z2) represents the joint probability distribution function of z1 and z2;

[0021] When the overall node similarity MI is higher than a preset similarity threshold, the webpage type is determined to be a list webpage, otherwise it is a text webpage.

[0022] As a further improvement to the above solution, the method for extracting the first key information from the HTML text includes:

[0023] The node where the target word appears in the HTML text is used as a list row node, and the hyperlink, title and publishing time in the list row node are extracted as the first key information.

[0024] As a further improvement to the above solution, the second key information extraction method in HTML text includes:

[0025] Based on the content information encoding of the node, it is determined whether the Chinese character density and punctuation density of each node in the HTML text are higher than their corresponding preset density thresholds. If so, the node is used as the main text node, the main text content in the main text node is extracted, and the title and release time are extracted from the node before the main text node, which are used as the second key information.

[0026] As a further improvement to the above solution, before performing the feature analysis and encoding, the HTML text is also pre-processed to remove specific tags; the types of specific tags include <input> Label, <script>标签、<style>标签和<iframe>标签。

[0027] 本发明还公开一种网页信息提取系统，应用如上所述的一种网页信息提取方法；所述网页信息提取系统包括：

[0028] 信息编码模块，用于获取待提取信息网页的HTML文本，分别从位置信息、结构信息和内容信息三个维度对HTML文本的每个节点进行特征分析并编码，从而构建HTML文本的节点编码向量集合；以及

[0029] 分类提取模块，用于根据所述节点编码向量集合判断HTML文本的网页类型，在网页类型为列表网页时提取HTML文本中的第一关键信息，在网页类型为正文网页时提取HTML文本中的第二关键信息。

[0030] 与现有技术相比，本发明的有益效果是：

[0031] 1、本发明公开的基于多维度特征分析与建模的网页信息提取方法，能够更准确地识别和提取网页中的关键信息。通过综合考虑节点的位置信息、结构信息和内容信息，能够全面刻画节点的特征，从而提高网页信息提取的准确性和鲁棒性。相比于现有技术中仅依靠特定模板或规则进行信息提取的方法，本发明能够更好地适应不同结构和风格的网页，减少误提取和漏提取的情况。

[0032] 另外，本发明采用的互信息相似度计算方式，通过结构信息编码计算节点相似程度，这种方法相比现有技术中常用的基于距离或简单内容匹配的相似度计算方法，能够更全面地考虑节点的位置和结构特征，从而更准确地评估节点之间的相似性。这种方法不仅能够有效处理复杂网页结构，还能够提升信息提取的精度和鲁棒性。

[0033] 2、本发明采用Xpath路径编码方法对节点的位置信息进行编码，能够更精确地描述节点在HTML文档中的位置关系。通过将标签和索引进行编码，本发明能够有效地表示节点之间的相对位置，从而为后续的特征分析和信息提取提供有力支持。并通过将节点与其他节点之间的编码的差集来代表该节点的结构信息，相比于现有技术中常用的基于标签或类名的位置信息描述方法，本发明能够更准确地反映节点的实际位置，并能提供更准确的结构信息分析，进一步提高网页信息提取的准确性。

[0034] 3、本发明采用基于内容密度的计算来提取信息，可以从网页内容中识别出密度较高的关键信息，与现有技术常用的基于固定规则或模板的信息提取方法相比，能够更灵活地适应不同网页的变化和内容结构的多样性。这种基于内容密度的计算方式不受特定布局或格式的限制，有效减少了误提取和漏提取的情况，显著提高了信息提取的准确性和实用性。

[0035] 4、本发明公开的网页信息提取系统，其通过应用上述方法，能产生与上述方法相同的有益效果，在此不再赘述。附图说明

[0036] 图1为本发明实施例1中的基于多维度特征分析与建模的网页信息提取方法的流程图。

[0037] 图2为本发明实施例1中的构建HTML文本的节点编码向量集合方法的流程图。

[0038] 图3为本发明实施例1中的判断网页类型并提取关键信息方法的流程图。

[0039] 图4为本发明实施例2中的基于多维度特征分析与建模的网页信息提取系统的框架图。具体实施方式

[0040] 下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。

[0041] 实施例1

[0042] 请参阅图1，本实施例提供一种基于多维度特征分析与建模的网页信息提取方法，可包括以下步骤即S1～S3。

[0043] S1.获取待提取网页信息的HTML文本，分别从位置信息、结构信息和内容信息三个维度对HTML文本的每个节点进行特征分析并编码，从而构建HTML文本的节点编码向量集合。

[0044] 在一些实施例中，在进行所述特征分析并编码之前，还对HTML文本进行数据预处理，剔除特定标签；所述特定标签的种类包括<input>标签、<script>标签、<style>标签和<iframe>标签。这些标签包含的内容对于最终用户来说不是直接可见的，或者是用于网页的布局和样式，而不是内容传递。

[0045] HTML文本的节点特征向量集合表示为XNode，XNode＝{X1,X2,…,Xi,…,XN}；式中，Xi表示第i个节点的特征向量，i∈[1,N]，N为HTML文本中的节点总数；表示第i个节点的位置信息编码；表示第i个节点的结构信息编码；表示第i个节点的内容信息编码。

[0046] 请参阅图2，本实施例中，构建HTML文本的节点编码向量集合可包括步骤S11～S13。

[0047] S11.构建节点的位置信息编码：

[0048] 获取当前节点的Xpath(XML路径语言)路径。本实施例中，节点的Xpath路径表达式示例如下：

[0049] / html / body / div[2] / ul[2] / li

[13]

[0050] 这个Xpath路径表达式表示HTML文本中body标签下的第二个div标签下的无序列表(ul)中的第一个列表项(li)。

[0051] 对于Xpath路径中的标签部分使用枚举映射方式进行编码，对于Xpath路径中的索引部分直接使用索引序号进行编码，从而获取当前节点基于Xpath路径的位置信息编码。例如，将html转换为1，body转换为2，div转换为3，ul转换为4，li转换为5。索引序号保持不变，例如，[2]保持为2，

[13] 保持为13。这样，上述Xpath路径表达式可以被编码为即构建出节点的位置信息编码。

[0052] 特别地，为保证确定性，通常只对常见的标签进行映射，如果超出已知标签范围，则利用99统一代替，表示为特殊标签。

[0053] S12.构建节点的结构信息编码：

[0054] 获取当前节点的所有子节点，依次将各个子节点基于Xpath路径的位置信息编码减去当前节点基于Xpath路径的位置信息编码，将各减法结果汇总以形成当前节点的结构信息编码。

[0055] 例如，HTML文本中有四个节点，其中有节点i的Xpath路径表达式为 / html / body / div[2] / ，它的位置信息编码为

[0056] 第一个子节点：假设这个子节点是一个div标签，没有指定索引，那么它的Xpath路径表达式可以是 / html / body / div[2] / div，对应的位置信息编码为：

[0057] 第二个子节点：假设这个子节点是一个ul标签，没有指定索引，那么它的Xpath路径表达式可以是 / html / body / div[2] / ul。对应的位置信息编码为

[0058] 第三个子节点：假设这个子节点第一个子节点的子节点，是一个p标签，并且是第一个p标签，那么它的Xpath路径表达式是 / html / body / div[2] / div / p[1]。

[0059] 对应的位置信息编码为

[0060] 因此，依次用第一、第二、第三个子节点的位置信息编码减去节点i的位置信息编码，从而得到节点i的结构信息编码

[0061] S13.构建节点的内容信息编码：

[0062] 统计当前节点的字符总数qA和子节点总数qB，分析当前节点中的超链接数量q1、中文字符数量q2以及标点符号数量q3，并计算当前节点的超链接密度ρ1、中文字符密度ρ2和标点符号密度ρ3，由qA、qB、q1、q2、q3、ρ1、ρ2、ρ3构建当前节点的内容信息编码；其中，ρ1＝q1 / qB；ρ2＝q2 / qA；ρ3＝q3 / qA。

[0063] 例如，对于表1所示的HTML文本代码示例：

[0064] 表1.HTML文本代码示例

[0065]

[0066] 首先统计当前HTML节点的字符总数，该节点包含73个字符，3个子节点。接着，该节点共有4个标点符号，包括逗号、句号和冒号。然后统计超链接的数量，发现该节点中包含0个超链接。此外，还统计了中文字符的数量，该节点包含54个中文字符。

[0067] 最后，标点符号与字符总数的比值为4 / 73，超链接与子节点总数的比值为0 / 3，中文字符与字符总数的比值为54 / 73。因此，上述节点的内容信息编码为

[0068] S2.根据所述节点编码向量集合判断HTML文本的网页类型，在网页类型为列表网页时提取HTML文本中的第一关键信息，在网页类型为正文网页时提取HTML文本中的第二关键信息。

[0069] 请参阅图3，本实施例中，可通过互信息计算节点之间的相似度，从而计算出HTML文本的总体节点相似度MI，计算公式为：

[0070]

[0071] 式中，Zi'和Zj'分别表示HTML根节点结构信息编码中的两个向量；I(·)表示两个向量之间的相似度计算函数；z1表示向量Zi'中的一个元素，z2表示向量Zj'中的一个元素；p(z1)表示z1的边缘概率分布函数；p(z2)表示z2的边缘概率分布函数；p(z1,z2)表示z1和z2的联合概率分布函数。

[0072] 当所述总体节点相似度MI高于预设的相似度阈值时，判断网页类型为列表网页，反之则为正文网页。

[0073] 在其他实施例中，还可以从所述节点编码向量集合中提取出分类所需特征，所述分类所需特征包括：HTML文本的超链接密度H和内容密度C，从而计算HTML文本的网页类型评分Score：

[0074] Score＝wH×H+wC×C

[0075] 式中，H＝q1f / N；C＝(q2f+q3f) / qAf；q1f为HTML文本中所有节点的超链接总数；q2f和q3f分别为HTML文本中所有节点的中文字符总数和标点符号总数，qAf为HTML文本中所有节点的字符总数；wH和wC均为权重，且满足wH＞0，wC＜0。

[0076] 当所述网页类型评分Score高于预设的分数阈值时，判断网页类型为列表网页，反之则为正文网页。

[0077] 例如下表2所示的列表网页类型的HTML文本：

[0078] 表2.HTML文本代码示例

[0079]

[0080]

[0081] 该HTML文本包括4个节点，该HTML文本的总体节点相似度为0.8664，相似度阈值为0.7。因此，该HTML文本是列表网页类型。

[0082] 或者也可以通过提取出其中的更深层次特征：超链接密度：3 / 11，内容密度：9 / 39。权重wH＝3.667，wC＝-1.2684，网页类型评分Score＝0.7071，内容阈值设定为0.5。因此也可以判断出该HTML文本是列表网页类型。由于网页类型复杂，具体权重和阈值需根据网页情况进行自适应更改。

[0083] 对于正文网页，如表3所示。

[0084] 表3.HTML文本代码示例

[0085]

[0086]

[0087] 可以计算出超链接密度：1 / 12，内容密度：148 / 156，网页类型评分Score＝-0.898。因此可以判断出HTML文本是正文网页类型。

[0088] 在判断出网页类型后，若为列表网页，HTML文本中的第一关键信息提取方法包括S21。

[0089] S21.将HTML文本中出现目标字样的节点作为列表行节点，提取所述列表行节点中的超链接、标题和发布时间作为第一关键信息。

[0090] 需要说明的是，目标字样通常包含能够标识列表行关键信息的HTML元素或属性。在大多数情况下，这些目标字样可能包括包含超链接的标签(通常带有href属性，指向详细内容页面)，title属性(常用于存储完整的标题信息)，包含文本内容的元素(通常是标签内的文本，代表显示的标题)，表示发布时间的元素(如标签，可能带有特定的class属性，如'publishtime'或'date'等)，以及包裹整个列表项的标签。这些目标字样在不同网站中可能有细微的变化，但基本结构通常保持一致。

[0091] 例如：<a href＝'链接地址'title＝'完整标题'>显示标题<span class＝'publishtime'>发布时间< / li。

[0092] 若网页类型为正文网页，HTML文本中的第二关键信息提取方法包括S22。

[0093] S22.根据节点的内容信息编码，判断HTML文本中的每个节点的中文字符密度和标点符号密度是否均高于各自对应的预设密度阈值，是则将该节点作为正文节点，提取所述正文节点中的正文内容，并在所述正文节点之前的节点中提取标题和发布时间，据此作为第二关键信息。

[0094] 需要说明的是，正文页的内容比较集中，并且很少出现超链接，而会出现大量几种的文字和标点符号。因此选择出其中内容密度大的节点，通过内容信息中的中文字符密度、标点符号密度、超链接密度同时进行判断，得到正文页的正文内容节点。正文页中的标题和发布时间，通常在正文内容的上方，并距离较近，所以可以通过位置信息减少搜索的范围，再进行精确的信息提取。在一些实施例中，中文字符密度和标点符号密度对应的预设密度阈值，可以根据网页的具体类型、领域特征和内容特点而定。例如，新闻类网页和学术论文类网页可能需要设置不同的密度阈值。此外，还可以通过机器学习算法，基于大量样本数据，自动学习和调整这些阈值，以适应不同类型网页的特点。

[0095] S3.将步骤S2中提取的关键信息进行分类存储，以便于后续的数据分析。

[0096] 实施例2

[0097] 请参阅图4，本实施例提供一种基于多维度特征分析与建模的网页信息提取系统10，该系统可以应用实施例1中基于多维度特征分析与建模的网页信息提取方法；网页信息提取系统10包括信息编码模块101以及分类提取模块102。

[0098] 信息编码模块101用于获取待提取信息网页的HTML文本，分别从位置信息、结构信息和内容信息三个维度对HTML文本的每个节点进行特征分析并编码，从而构建HTML文本的节点编码向量集合。

[0099] 分类提取模块102用于根据所述节点编码向量集合判断HTML文本的网页类型，在网页类型为列表网页时提取HTML文本中的第一关键信息，在网页类型为正文网页时提取HTML文本中的第二关键信息。

[0100] 以上所述，仅为本发明较佳的具体实施方式，但本发明的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本发明揭露的技术范围内，根据本发明的技术方案及其发明构思加以等同替换或改变，都应涵盖在本发明的保护范围之内。< / script>

Claims

1. A web page information extraction method, characterized in that: include: Obtain the HTML text of the web page information to be extracted, perform feature analysis and encoding on each node of the HTML text from three dimensions: position information, structure information, and content information, and thus construct a node encoding vector set of the HTML text; Determining the webpage type of the HTML text according to the node coding vector set, extracting first key information from the HTML text when the webpage type is a list webpage, and extracting second key information from the HTML text when the webpage type is a text webpage; Among them, the node feature vector set of HTML text is represented as X Node , X Node ={X1,X2,…,X i ,…,X N }; where X i Represents the feature vector of the i-th node, i∈[1,N], N is the total number of nodes in the HTML text; Indicates the location information encoding of the i-th node; Represents the structural information encoding of the i-th node; Represents the content information encoding of the i-th node; The construction method of node location information encoding is as follows: Get the XPath path of the current node. Encode the label part of the XPath path using enumeration mapping, and directly encode the index part of the XPath path using the index number, thereby obtaining the location information encoding of the current node based on the XPath path. The construction method of the node structure information encoding is as follows: Get all child nodes of the current node, subtract the location information code of the current node based on the XPath path from the location information code of each child node based on the XPath path, and aggregate the subtraction results to form the structure information code of the current node; The node content information encoding construction method is as follows: Count the total number of characters q in the current node A and the total number of child nodes q B , analyze the number of hyperlinks q1, the number of Chinese characters q2, and the number of punctuation marks q3 in the current node, and calculate the hyperlink density ρ1, Chinese character density ρ2, and punctuation mark density ρ3 of the current node. A ,q B , q1, q2, q3, ρ1, ρ2, ρ3 construct the content information encoding of the current node; where ρ1=q1 / q B ;ρ2=q2 / q A ;ρ3=q3 / q A ; The determining the webpage type of the HTML text according to the node coding vector set includes: The similarity between nodes is calculated through mutual information, so as to calculate the overall node similarity MI of HTML text. The calculation formula is: Where Z i' and Z j' They represent two vectors in the HTML root node structure information encoding; I(·) represents the similarity calculation function between the two vectors; z1 represents the vector Z i' An element in , z2 represents the vector Z j' An element in ; p(z1) represents the marginal probability distribution function of z1; p(z2) represents the marginal probability distribution function of z2; p(z1,z2) represents the joint probability distribution function of z1 and z2; When the overall node similarity MI is higher than a preset similarity threshold, the webpage type is determined to be a list webpage, otherwise it is a text webpage.

2. A web page information extraction method according to claim 1, characterized in that: The first key information extraction method in HTML text includes: The node where the target word appears in the HTML text is used as a list row node, and the hyperlink, title and publishing time in the list row node are extracted as the first key information.

3. A web page information extraction method according to claim 1, characterized in that: The second key information extraction method in HTML text includes: Based on the content information encoding of the node, it is determined whether the Chinese character density and punctuation density of each node in the HTML text are higher than their corresponding preset density thresholds. If so, the node is used as the main text node, the main text content in the main text node is extracted, and the title and release time are extracted from the node before the main text node, which are used as the second key information.

4. A web page information extraction method according to claim 1, characterized in that: Before performing the feature analysis and encoding, the HTML text is also pre-processed to remove specific tags; the types of specific tags include <input> Label, <script>标签、<style>标签和<iframe>标签。5.一种网页信息提取系统，其特征在于，应用如权利要求1至4中任意一项所述的一种网页信息提取方法；所述网页信息提取系统包括：信息编码模块，用于获取待提取信息网页的HTML文本，分别从位置信息、结构信息和内容信息三个维度对HTML文本的每个节点进行特征分析并编码，从而构建HTML文本的节点编码向量集合；以及分类提取模块，用于根据所述节点编码向量集合判断HTML文本的网页类型，在网页类型为列表网页时提取HTML文本中的第一关键信息，在网页类型为正文网页时提取HTML文本中的第二关键信息。< / script>

Images