Methods, apparatus, and devices for identifying closely related web pages based on function and structure.

Abstract

This invention provides a method, apparatus, and device for identifying closely related web pages based on function and structure. The method includes: acquiring page information of two web pages to be identified; extracting functional features from the two web pages, inputting the functional feature vectors into a functional classification model to predict the functional labels of the two web pages; searching for a corresponding clustering model in a clustering model set based on the functional labels; inputting the structural feature vectors into the corresponding clustering model to predict the cluster to which each of the two web pages belongs, wherein the clusters and the functional labels constitute the functional and structural features of the web pages; and determining whether the two web pages are closely related pages based on their respective functions and structures. This invention obtains the function of a page through functional identification, and then identifies the structure through structural clustering based on the functional information, thereby effectively characterizing the implementation structure of web pages and improving the accuracy of identifying closely related web pages.

Description

Technical Field

[0001] This invention relates to the field of computer technology, and in particular to a method, apparatus, and device for identifying closely related web pages based on their function and structure. Background Technology

[0002] With the rise of open-source software and low-code development, web page development has become increasingly convenient. Many web pages are developed based on specific templates, resulting in many closely related pages with similar implementation structures (i.e., the logical structure of the web page source code). Identification of closely related pages plays an important role in areas such as code clone detection, web data extraction, and copyright protection.

[0003] However, due to the complexity of web page code, many code segments are related to style information such as appearance and layout, making direct representation methods like the DOM tree ineffective and posing challenges to representing the implementation structure of web pages. Therefore, improving the representation of the implementation structure of web pages is key to improving the accuracy of identifying closely related web pages. Summary of the Invention

[0004] In view of the above problems, embodiments of the present invention provide a method, apparatus, device, and medium for identifying closely related web pages based on function and structure, so as to overcome the above problems or at least partially solve the above problems.

[0005] A first aspect of this invention discloses a method for identifying closely related web pages based on function and structure, the method comprising:

[0006] Obtain page information for the two web pages to be identified;

[0007] Functional features are extracted from the two web pages to obtain functional feature vectors for each web page. The functional feature vectors are then input into a functional classification model to predict the functional labels of the two web pages.

[0008] Based on the functional labels, the corresponding clustering model is found in the clustering model set, and the structural features of the two web pages are extracted to obtain the structural feature vectors of the two web pages respectively.

[0009] The structural feature vector is input into the corresponding clustering model to predict the cluster to which each of the two web pages belongs. The cluster to which the web page belongs and the functional tags of the web page constitute the function and structure of the web page.

[0010] Based on the respective functions and structures of the two web pages, determine whether the two web pages are related pages.

[0011] Optionally, functional features are extracted from the two web pages to obtain functional feature vectors for each web page. These functional feature vectors are then input into a functional classification model to predict the functional labels of the two web pages. The functional classification model is obtained through pre-training, and the training of the functional classification model includes:

[0012] The training data is labeled according to a predefined set of functional categories to obtain a set of labeled web pages for model training;

[0013] The marked Web page set is subjected to feature engineering to obtain a functional feature matrix for each Web page in the marked Web page set, wherein the features in the functional feature matrix include at least the number of tags, text information, manual rules and semantic information;

[0014] The functional classification model is trained based on the feature matrix of each web page to obtain the functional classification model;

[0015] The method further includes:

[0016] When predicting the function of a web page, if the function label is incorrect, the label is corrected so that the function classification model can learn incrementally on the corrected data.

[0017] Optionally, a corresponding clustering model is searched in the clustering model set according to the functional label, wherein the clustering models in the clustering model set are obtained through pre-training, and the clustering model training includes:

[0018] The labeled set of web pages is divided into multiple web page sets according to function tags, and the web pages in each web page set have the same function tags;

[0019] For each web page in each set of web pages, feature engineering is performed to obtain the structural feature matrix of each web page. The clustering model is trained using the structural feature matrix to obtain a set of clustering models corresponding to multiple functional labels.

[0020] Optionally, the structural feature vector is input into the corresponding clustering model to predict the cluster to which the two web pages belong, including:

[0021] The cluster to which the web page belongs is obtained by comparing the center of all clusters in the clustering model with the structural feature vector of the web page.

[0022] Optionally, determining whether the two web pages are related pages based on their respective implementation structures includes:

[0023] If the two web pages have similar implementation structures, then the two web pages are determined to be closely related pages.

[0024] Optionally, based on the respective functions and structures of the two web pages, the method further includes determining whether the two web pages are related pages.

[0025] Obtain the semantic theme of each of the two web pages, where the semantic theme represents the content theme described by the web pages;

[0026] If the two web pages have similar functions and structures, and the two web pages have the same semantic theme, then the two web pages are determined to be closely related pages.

[0027] A second aspect of the present invention discloses a device for identifying closely related web pages based on function and structure, the device comprising:

[0028] The information acquisition module is used to acquire page information of the two web pages to be identified;

[0029] The function recognition module is used to extract function features from the two web pages, obtain function feature vectors for the two web pages respectively, and input the function feature vectors into the function classification model to predict the function labels of the two web pages.

[0030] The structural feature extraction module is used to find the corresponding clustering model in the clustering model set according to the functional label, extract the structural features of the two web pages, and obtain the structural feature vectors of the two web pages respectively.

[0031] The function and structure generation module is used to input the structural feature vector into the corresponding clustering model to predict the cluster to which the two web pages belong, and the cluster to which the web page belongs and the function label of the web page constitute the function and structure of the web page;

[0032] The judgment module is used to determine whether the two web pages are related pages based on their respective functions and structures.

[0033] Optionally, the function recognition module includes:

[0034] The first labeling module is used to label the training data according to a predefined set of functional categories to obtain a set of labeled web pages for model training;

[0035] The first feature processing module is used to perform feature engineering processing on the tagged Web page set to obtain a functional feature matrix for each Web page in the tagged Web page set, wherein the features in the functional feature matrix include at least the number of tags, text information, manual rules and semantic information;

[0036] The classification training module is used to train the functional classification model based on the feature matrix of each Web page to obtain the functional classification model;

[0037] The incremental learning module is used to correct labels when predicting the function of a web page if the function label is incorrect, so that the function classification model can perform incremental learning on the corrected data.

[0038] Optionally, the function and structure generation module includes:

[0039] The second tagging module is used to divide the tagged Web page set into multiple Web page sets according to functional tags, and the Web pages in each Web page set have the same functional tags;

[0040] The second feature processing module is used to perform feature engineering on each web page in each web page set to obtain a structural feature matrix for each web page, and to train a clustering model using the structural feature matrix to obtain a set of clustering models corresponding to multiple functional labels.

[0041] A third aspect of the present invention discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the close relative web page identification method based on function and structure as described in the first embodiment of the present invention.

[0042] The embodiments of the present invention have the following advantages:

[0043] In this embodiment of the invention, page information of two web pages to be identified is obtained. Functional features are extracted from the two web pages to obtain functional feature vectors for each. These feature vectors are then input into a functional classification model to predict the functional labels of the two web pages. Based on the functional labels, a corresponding clustering model is searched in a clustering model set. Structural features are extracted from the two web pages to obtain structural feature vectors for each. These structural feature vectors are then input into the corresponding clustering model to predict the cluster to which each web page belongs. The cluster to which a web page belongs, along with its functional label, constitutes the function and structure of the web page. Finally, based on the function and structure of each web page, it is determined whether the two web pages are related pages. In this embodiment, since one function of a web page corresponds to several specific implementation structures, the function of the web page can be used to reflect the implementation structure. First, functional identification of the web page is performed to obtain its functional label, thereby assisting in the identification of the implementation structure. After functional identification, only a limited number of implementation structures remain for each function. At this point, clustering is used to distinguish these implementation structures. The scenario faced by clustering is not complex, so the clustering effect is better. Therefore, function-based structural clustering methods can more effectively characterize the implementation structure of web pages, and thus accurately identify closely related web pages based on the functions and structure of the web page. Attached Figure Description

[0044] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a schematic diagram illustrating the technical concept of a function-based structural clustering method provided in an embodiment of the present invention;

[0046] Figure 2 This is a flowchart of a function-based structural clustering method provided in an embodiment of the present invention;

[0047] Figure 3 This is a flowchart illustrating the steps of a method for identifying closely related web pages based on function and structure, provided in an embodiment of the present invention.

[0048] Figure 4 This is a schematic diagram of a login page and its corresponding main structure's tag tree, provided by an embodiment of the present invention.

[0049] Figure 5This is a flowchart of a function identification method provided in an embodiment of the present invention;

[0050] Figure 6 This is a flowchart of a structural clustering method provided in an embodiment of the present invention;

[0051] Figure 7 This is a schematic diagram of a close relative web page recognition device based on function and structure provided in an embodiment of the present invention. Detailed Implementation

[0052] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0053] Regarding the accurate representation of the implementation structure of a web page, there are three main problems: 1) How to represent the implementation structure of a page: Web page code is very complex, with many code segments related to appearance and layout style information. This makes direct representation methods like the DOM tree ineffective, posing a challenge to the representation of the structure. The applicant discovered that there is a close relationship between the page's functionality and its implementation structure, and thus the structure can be indirectly represented through functionality; 2) How to identify the page's functionality: Page functionality is intuitive for humans, but not so intuitive for machines. Therefore, how to make machines understand the page's functionality is also a major problem; 3) How to map functionality to structure: This requires the recognition of structure. Therefore, after obtaining the page's functional information, it is also necessary to consider how to map the functionality to the structure. Finding the corresponding mapping is not easy for machines.

[0054] To address the aforementioned issues, the applicant proposes the following technical concept: Based on the observation of a close relationship between the functions and implementation structures of web pages, this method uses the functional information of web pages to assist in the identification of page structures. The method first identifies the functions of the web page to obtain its functionalities, and then performs structural clustering based on the functional information to identify the structure. Since after identifying the functions of the web page, only a limited number of implementation structures remain for each function, clustering is then used to distinguish these implementation structures. Because the clustering scenario is not complex at this point, the clustering effect is better. Figure 1As shown, the function-based structural clustering method can solve the problem of structural representation between web pages mentioned above. Functions can effectively represent the implementation structure of web pages, function identification methods can effectively identify the functions of web pages, and structural clustering can effectively map functions to structures.

[0055] Specifically, this method mainly includes two parts: function identification and structural clustering, such as... Figure 2 As shown, the process first extracts functional features from the web page to generate feature vectors. These feature vectors are then input into a functional classification model for prediction, yielding the web page's functional label. Next, the structural clustering module searches for the corresponding clustering model based on the functional label. It then extracts structural features from the web page to obtain feature vectors, which are input into a prediction model to determine the cluster to which the web page belongs. The cluster to which the web page belongs and the functional label together generate the final function and structure. Finally, based on this function and structure, two web pages are identified to determine if they are related pages.

[0056] Based on the above technical concept, embodiments of the present invention provide a method for identifying closely related web pages based on function and structure, referring to... Figure 3 , Figure 3 A flowchart illustrating the steps of a method for identifying closely related web pages based on function and structure, as provided in this embodiment of the invention, is shown below. Figure 3 As shown, the method includes:

[0057] Step S301: Obtain page information for the two web pages to be identified.

[0058] Algorithms are used to retrieve page information from two web pages. These web pages consist of HTML, CSS, and JavaScript code, many of which are related to layout, appearance, and interactive behavior. For example, tags contain semantic information about the page. <style>标签中涉及的是样式信息，与文档在浏览器上的展示相关联，<script>标签中包含的是脚本语言，与文档在浏览器上的交互行为相关联。

[0059] 步骤S302：对所述两个Web页面进行功能特征提取，分别得到所述两个Web页面的功能特征向量，分别将所述功能特征向量输入功能分类模型，预测所述两个Web页面的功能标签。

[0060] 在本实施例中，通过页面的功能信息辅助对Web页面结构的识别，为此需要能够识别页面的功能。首先对Web页面进行功能特征提，得到能够反映出Web页面功能的功能特征向量，即通过该功能特征向量就能够比较好地区分不同的功能Web页面，例如标签数量、文本信息、人工规则以及语义信息等功能特征向量，最后将功能特征向量输入到预先训练好的功能分类模型中进行功能预测，进而得到两个Web页面的功能标签。

[0061] 步骤S303：根据所述功能标签在聚类模型集合中寻找对应的聚类模型，对所述两个Web页面进行结构特征提取，分别得到所述两个Web页面的结构特征向量。

[0062] 每个Web页面都会有一个对应的主体结构，例如，如图4中(a)所示的包含登录功能的Web页面，其通常会包含(b)所示的主体结构，因为登录需要用户提供相应的用户名与密码，所以必然存在一个表单，允许用户输入用户名密码并提交到服务器。因此，我们可以使用Web页面的功能来反映Web页面的主体结构，可以通过了解Web页面的功能来间接地掌握Web页面的主体结构。

[0063] Web页面的每种功能通常会对应多种特定的实主体结构，例如，对于列表内容而言，其可以对应如下所示的和标签组成的结构，这是标准的列表用法；也可能对应的标签组成的结构，这通常用于与CSS结合起来实现自定义的布局等样式表现。因此，在对Web页面的功能进行识别后，根据得到的功能标签在聚类模型集合中找到该功能对应的聚类模型，并将该对应的聚类模型用于后续结构的预测。

[0064]

[0065]

[0066] 步骤S304：将所述结构特征向量输入对应的聚类模型，预测所述两个Web页面各自所属的簇，所述Web页面所属的簇与所述Web页面的功能标签构成所述Web页面的功能和结构。

[0067] 聚类算法采用的结构特征向量主要是Web页面的标签频率，因为聚类是在功能识别的基础上进行的，通过功能信息已经将大部分结构都排除了，只剩下了同一种功能对应的少数几种结构，在这种情况下通过页面的标签频率就可以比较好地区分出各个结构。例如，对于列表内容而言，其可以对应如上所示的和标签组成的结构，也可能对应标签组成的结构，这两种结构通过标签频率可以很好地辨别出来。

[0068] 在本实施例中，将Web页面的结构特征相似输入到对应的聚类模型中，得到Web页面所属的簇，其中，一种簇对应于一种Web页面的主体结构，即不同簇的Web页面主体结构也不同，进而根据Web页面所属的簇与Web页面的功能标签构成的功能和结构，能够很好的表征Web页面的实现结构，即通过结构功能能够较为准确的识别近亲Web页面。

[0069] 步骤S305：根据所述两个Web页面各自的实现结构，判断所述两个Web页面是否为近亲页面。

[0070] 由于近亲Web页面的实现结构相似，通过上述步骤得到的功能和结构能够准确的反映出对应Web页面实现结构，即准确反映Web页面源代码的实现逻辑。因此通过对两个Web页面的功能和结构进行比较，就能判断两个Web页面是否为近亲页面。

[0071] 在本实施例中，由于Web页面中一种功能对应特定的几种实现结构，因此可使用Web页面的功能来反映实现结构，通过对页面进行功能识别得到其功能标签从而辅助对实现结构的识别。在功能识别之后，每种功能所对应的实现结构只剩下有限的几种，再使用聚类将这几种实现结构区分开，因为此时聚类所面临的场景并不复杂，所以聚类的效果会比较好。因此基于功能的结构聚类方法可以有效地表征Web页面的实现结构，进而基于该Web页面功能和结构进行近亲Web页面识别的准确度更高。

[0072] 在一种实施例中，对所述两个Web页面进行功能特征提取，分别得到所述两个Web页面的功能特征向量，分别将所述功能特征向量输入功能分类模型，预测所述两个Web页面的功能标签；所述功能分类模型是通过预先训练而得到的，所述功能分类模型训练包括：

[0073] 根据预先定义的功能类别集合对训练数据进行标记，得到用于模型训练的标记Web页面集；

[0074] 对所述标记Web页面集进行特征工程处理，得到所述标记Web页面集中每个Web页面的功能特征矩阵，其中，所述功能特征矩阵中的特征至少包括标签数量、文本信息、人工规则以及语义信息；

[0075] 根据所述每个Web页面的特征矩阵对功能分类模型进行训练，得到功能分类模型；

[0076] 所述方法还包括：

[0077] 在对Web页面进行功能预测时，若功能标签是错误，则纠正标签以使功能分类模型在纠正后的数据上进行增量学习。

[0078] 在本实施例中，通过页面的功能信息辅助对页面结构的识别，为此需要能够识别页面的功能，而页面的功能对机器而言并不直观，让机器能理解功能是有难度的。为此，利用使用机器学习来解决这个问题，期望计算机可以从数据中自动分析学习从而进行对功能的预测，进而使用机器学习中的分类模型来学习预测页面的功能。

[0079] 具体地，如图5所示，功能识别分为两个阶段：离线训练功能模型训练阶段与在线功能预测阶段。离线阶段是指对利用Web页面集对功能分类模型进行训练，具体地包括：定义类别集合、搜寻并标记Web页面集、进行特征工程出来以及训练模型，进而训练出可预测页面功能的分类模型。

[0080] 首先，定义功能类别的集合，所述的功能至少包括：用户密码登录、普通搜索、高级搜索、找回密码、填表、记录表、信息表、列状内容、列状目录、纯文本与文本图片等类别。然后，需要获取数据集来训练模型。利用算法爬取网络上各种网站的Web页面，并进行去重等过滤操作，然后对这些Web页面进行人工标记，最终得到多个已标记Web页面组成的数据集。之后，对Web页面集合进行特征工程处理，找出最能代表Web页面功能的特征。本文选取的特征主要分为标签数量、文本信息、人工规则以及语义信息等四种特征。

[0081] 标签数量是指标签出现的次数，标签数量可以比较好地区分不同的功能页面。例如，<h>、<form>、<input>、、、、、、<button>、<textarea>、、、和<article>等标签的数量。<h>标签是标题标签，它的出现次数可以有效地区分很多类别，比如只出现一次的话很有可能是与文本相关的类别。<form>标签是表单标签，用于向服务器传输数据，包含<form>标签的页面很可能是登录、搜索等表单相关的类别。<input>标签是输入标签，可以让用户输入信息，它有很多种不同的类型，这些不同类型<input>标签的数量可以非常有效地区分很多类别，比如有密码输入标签则很可能是登录相关的类别。标签是图像标签，会向页面嵌入图像，根据它的出现次数可以区分很多与图像有关的类别。标签是超链接标签，可以链接到指定的url地址，它可以链接到其他Web页面，也可以链接到本页面其他位置，根据它的出现次数可以区分与跳转链接相关的类别。、、标签是与列表相关的标签，如果它出现很多次，那么很可能是列状内容或列状目录等类别。<button>标签是按钮标签，可以为用户提供交互行为，因此它的出现次数可以区分很多与互动相关的类别。<textarea>标签可以让用户输入多行文本，它的出现表示很可能是评论等类别。、、是与表格相关的标签，根据它的出现次数可以区分很多与表格有关的类别。<article>标签是文本标签，表示它里面的内容是文章，具有清晰的语义信息，它的出现表示很可能是与文章相关的类别。

[0082] 文本信息是指标签内的文本数量以及文本密度等信息，文本信息也可以较好地区分不同的功能页面。例如<form>、、和等标签的文本数量，<form>标签内的文本数量可以很好地区分与表单相关的类别，、和标签内文本的数量可以很好地区分与列表相关的类别。我们也关注和等标签的文本密度，标签内文本密度大的话很可能是与文章相关的类别，标签内文本密度大的话很可能是列状内容等类别。

[0083] 人工规则是指人为指定的一些对确定页面功能有帮助的规则，可以非常有效地区分不同的功能页面。比如说<form>标签内是否包含类似"email”、"邮箱”、"phone”、"手机”、"user name”、"用户名”、"captcha”和"验证码”等文字，如果包含这些文字之一的话页面大概率会是与登录相关的类别。

[0084] 语义信息是指标签的属性或者文本所表示的语义内容，可以帮助区分不同的功能页面。例如<form>、<input>、和等标签的class和id属性的语义信息，其中class属性是类名，id属性是唯一标识。程序员在取class或id等变量名时通常会赋予语义信息，比如说对于登录表单来说，通常其class或者id属性会有"login”这样的文字，因此我们从这些属性内提取出语义信息可以帮助区分类别。

[0085] 在完成特征工程处理后，利用分类模型进行训练，同时为使算法模型可以持续学习新数据以提升表现效果，为此需要实现增量学习。而使用随机梯度下降方法训练的SVM模型可以支持增量学习，因此在本实施例中，使用基于随机梯度下降的支持向量机模型作为功能识别模型。

[0086] 在线阶段是指对Web页面进行功能预测，当在模型中输入一个Web页面，对该页面进行特征提取得到功能特征向量，再将功能特征向量输入在离线阶段训练好的功能分类模型，得到预测的功能标签。其中，若发现功能标签是错误的，可以纠正标签以使功能分类模型在纠正后的数据上进行增量学习。

[0087] 在一种实施例中，根据所述功能标签在聚类模型集合中寻找对应的聚类模型，其中，所述的聚类模型集合中的聚类模型是通过预先训练得到的，所述聚类模型训练包括：

[0088] 将带标记的Web页面集按照功能标签划分为多个Web页面集，每个所述Web页面集内的Web页面具有相同的功能标签；

[0089] 对每个所述Web页面集中的每个Web页面进行特征工程处理，得到所述每个Web页面的结构特征矩阵，利用所述结构特征矩阵对聚类模型进行训练，得到多个对应功能标签的聚类模型集合。

[0090] 具体地，如图6所示，对结构聚类模型进行训练时，将带标记的Web页面集按照功能标签划分为多个Web页面集，其中每个Web页面集合内的Web页面都有相同的功能标签，再对每个Web页面集合进行特征工程处理得到特征矩阵，并基于特征矩阵对聚类模型进行训练，最终得到多个对应功能标签的聚类模型集合。

[0091] 聚类的目的是在同一种功能的Web页面集合下进一步划分出同一种主体结构的Web页面集合。聚类会产生多个簇，属于同一簇的Web页面会拥有同一种主体结构，不同簇的Web页面主体结构也不同。

[0092] 在对聚类模型进行训练时，聚类将一个功能页面集合按照主体结构划分为多个簇，其输入是功能相同的Web页面构成的集合，输出是多个簇，其中每个簇由主体结构相同的Web页面构成。当在进行聚类预测时，聚类将预测一个Web页面所属的簇，其输入是单个Web页面，输出是该Web页面所属的簇。在本实施例中，使用K均值算法进行聚类，该算法会为每个类簇给出中心点，将新数据与每个类簇的中心比较，新数据属于其中距离最近的类簇。

[0093] 在对Web页面进行聚类预测时，输入一个Web页面，对该Web页面进行功能识别得到功能标签，再对Web页面进行特征提取得到特征向量，然后依据页面的功能标签找到对应的聚类模型，再将聚类模型中的所有簇的中心与页面的特征向量比较，得到该Web页面所属的簇，最终的功能和结构由功能标签与Web页面所属的簇名共同构成。

[0094] 在一种实施例中，将所述结构特征向量输入对应的聚类模型，预测所述两个Web页面各自所属的簇，包括：

[0095] 将所述聚类模型中的所有簇的中心与所述Web页面的结构特征向量比较，得到该Web页面所属的簇。

[0096] 在本实施例中，在对Web页面进行聚类预测时，将Web页面的结构特征向量输入到已经训练好的聚类模型，将聚类模型中的所有簇的中心与预测Web页面的特征向量比较，进而得到该Web页面所属的簇，最后Web页面所属的簇与功能标签共同构成功能和结构，并将该功能和结构用于Web页面的近亲识别。

[0097] 在一种的实施例中，根据所述两个Web页面各自的功能和结构，判断所述两个Web页面是否为近亲页面，包括：

[0098] 若所述两个Web页面的功能和结构相似，则判定所述两个Web页面为近亲页面。

[0099] 通过上述步骤得到功能和结构能够准确的反映Web页面源代码的逻辑结构，在本实施例中，通过比较两个Web页面的功能和结构，若两个Web页面的功能和结构相似，则判断这两个Web页面为近亲页面。

[0100] 在一种优选的实施例，根据所述两个Web页面各自的实现结构，判断所述两个Web页面是否为近亲页面，所述方法还包括：

[0101] 获取所述两个Web页面各自的语义主题，所述语义主题表征Web页面所描述的内容主题；

[0102] 若所述两个Web页面的实现结构相似，且所述两个Web页面的语义主题相同，则判断所述两个Web页面为近亲页面页。

[0103] 很多近亲Web页面的不仅实现结构相似，其Web页面的内容也相似，即Web的语义主题也相似，因此可以通过比较两个Web页面的语义主题和实现结构来判断两个Web页面是否为近亲页面。通过内容理解算法来提取Web页面的语义主题，所述语义主题表征Web页面所描述的内容主题，若两个Web页面的语义主题相似，且Web页面的功能和结构相同，就判断这两个Web页面为近亲页面。

[0104] 在本实施例中，通过利用Web页面的语义主题和功能和结构这两种信息同时来判断Web页面是否为近亲页面，进一步提高了近亲页面识别的准确性，

[0105] 如图7所示，图7为本发明实施例提供的一种基于功能和结构的近亲Web页面识别装置的结构示意图。如图7所示，所述装置具体可以包括以下模块：

[0106] 信息获取模块，用于获取待识别的两个Web页面的页面信息；

[0107] 功能识别模块，用于对所述两个Web页面进行功能特征提取，分别得到所述两个Web页面的功能特征向量，分别将所述功能特征向量输入功能分类模型，预测所述两个Web页面的功能标签；

[0108] 结构特征提取模块，用于根据所述功能标签在聚类模型集合中寻找对应的聚类模型，对所述两个Web页面进行结构特征提取，分别得到所述两个Web页面的结构特征向量；

[0109] 功能和结构生成模块，用于将所述结构特征向量输入对应的聚类模型，预测所述两个Web页面各自所属的簇，所述Web页面所属的簇与所述Web页面的功能标签构成所述Web页面的功能和结构；

[0110] 判断模块，用于根据所述两个Web页面各自的功能和结构，判断所述两个Web页面是否为近亲页面。

[0111] 在一种可选的实施例中，所述功能识别模块，包括：

[0112] 第一标记模块，用于根据预先定义的功能类别集合对训练数据进行标记，得到用于模型训练的标记Web页面集；

[0113] 第一特征处理模块，用于对所述标记Web页面集进行特征工程处理，得到所述标记Web页面集中每个Web页面的功能特征矩阵，其中，所述功能特征矩阵中的特征至少包括标签数量、文本信息、人工规则以及语义信息；

[0114] 分类训练模块，用于根据所述每个Web页面的特征矩阵对功能分类模型进行训练，得到功能分类模型；

[0115] 增量学习模块，用于在对Web页面进行功能预测时，若功能标签是错误，则纠正标签以使功能分类模型在纠正后的数据上进行增量学习。

[0116] 在一种可选的实施例中，所述功能和结构生成模块，包括：

[0117] 第二标记模块，用于将带标记的Web页面集按照功能标签划分为多个Web页面集，每个所述Web页面集内的Web页面具有相同的功能标签；

[0118] 第二特征处理模块，用于对每个所述Web页面集中的每个Web页面进行特征工程处理，得到所述每个Web页面的结构特征矩阵，利用所述结构特征矩阵对聚类模型进行训练，得到多个对应功能标签的聚类模型集合。

[0119] 本发明实施例还提供了一种电子设备，包括存储器、处理器及存储在存储器上并可在处理器上运行的计算机程序，所述处理器执行时实现如本发明实施例第一方面实施例所述的基于功能和结构的近亲Web页面识别方法。

[0120] 由于在本实施例中，Web页面的一种功能对应特定的几种实现结构，因此可使用Web页面的功能来反映实现结构，先通过对Web页面进行功能识别得到其功能标签从而辅助对实现结构的识别；在功能识别之后，每种功能所对应的实现结构只剩下有限的几种，此时再使用聚类将这几种实现结构区分开，聚类所面临的场景并不复杂，所以聚类的效果会更好。因此基于功能的结构聚类方法可以更有效地表征Web页面的实现结构，进而基于该Web页面功能和结构能够准确的识别近亲Web页面。

[0121] 本说明书中的各个实施例均采用递进的方式描述，每个实施例重点说明的都是与其他实施例的不同之处，各个实施例之间相同相似的部分互相参见即可。

[0122] 本发明实施例是参照根据本发明实施例的方法、装置、电子设备的流程图和 / 或方框图来描述的。应理解可由计算机程序指令实现流程图和 / 或方框图中的每一流程和 / 或方框、以及流程图和 / 或方框图中的流程和 / 或方框的结合。可提供这些计算机程序指令到通用计算机、专用计算机、嵌入式处理机或其他可编程数据处理终端设备的处理器以产生一个机器，使得通过计算机或其他可编程数据处理终端设备的处理器执行的指令产生用于实现在流程图一个流程或多个流程和 / 或方框图一个方框或多个方框中指定的功能的装置。

[0123] 这些计算机程序指令也可存储在能引导计算机或其他可编程数据处理终端设备以特定方式工作的计算机可读存储器中，使得存储在该计算机可读存储器中的指令产生包括指令装置的制造品，该指令装置实现在流程图一个流程或多个流程和 / 或方框图一个方框或多个方框中指定的功能。

[0124] 尽管已描述了本发明实施例的优选实施例，但本领域内的技术人员一旦得知了基本创造性概念，则可对这些实施例做出另外的变更和修改。所以，所附权利要求意欲解释为包括优选实施例以及落入本发明实施例范围的所有变更和修改。

[0125] 最后，还需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。而且，术语"包括”、"包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者终端设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者终端设备所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者终端设备中还存在另外的相同要素。

[0126] 以上对本发明所提供的一种功能和结构的近亲Web页面识别方法、装置和设备进行了详细介绍，本文中应用了具体个例对本发明的原理及实施方式进行了阐述，以上实施例的说明只是用于帮助理解本发明的方法及其核心思想；同时，对于本领域的一般技术人员，依据本发明的思想，在具体实施方式及应用范围上均会有改变之处，综上所述，本说明书内容不应理解为对本发明的限制。< / style>

Claims

1. A method for identifying closely related web pages based on function and structure, characterized in that, include: Obtain page information for the two web pages to be identified; Functional features are extracted from the two web pages to obtain functional feature vectors for each web page. The functional feature vectors are then input into a functional classification model to predict the functional labels of the two web pages. Based on the functional labels, the corresponding clustering model is found in the clustering model set, and the structural features of the two web pages are extracted to obtain the structural feature vectors of the two web pages respectively. The structural feature vector is input into the corresponding clustering model to predict the cluster to which the two web pages belong. The cluster to which the web page belongs and the functional tags of the web page constitute the structure and function of the web page. Based on the respective functions and structures of the two web pages, determine whether the two web pages are related pages.

2. The method according to claim 1, characterized in that, Functional features are extracted from the two web pages to obtain functional feature vectors for each web page. The functional feature vectors are then input into a functional classification model to predict the functional labels of the two web pages. The functional classification model is obtained through pre-training, and the training of the functional classification model includes: The training data is labeled according to a predefined set of functional categories to obtain a set of labeled web pages for model training; The marked Web page set is subjected to feature engineering to obtain a functional feature matrix for each Web page in the marked Web page set, wherein the features in the functional feature matrix include at least the number of tags, text information, manual rules and semantic information; The functional classification model is trained based on the feature matrix of each web page to obtain the functional classification model; The method further includes: When predicting the function of a web page, if the function label is incorrect, the label is corrected so that the function classification model can learn incrementally on the corrected data.

3. The method according to claim 1, characterized in that, The clustering model is searched in the clustering model set according to the functional label, wherein the clustering models in the clustering model set are obtained through pre-training, and the clustering model training includes: The labeled set of web pages is divided into multiple sets of web pages according to function tags, and the web pages in each set of web pages have the same function tags; For each web page in each set of web pages, feature engineering is performed to obtain the structural feature matrix of each web page. The clustering model is trained using the structural feature matrix to obtain a set of clustering models corresponding to multiple functional labels.

4. The method according to claim 1, characterized in that, The structural feature vectors are input into the corresponding clustering model to predict the cluster to which the two web pages belong, including: The cluster to which the web page belongs is obtained by comparing the center of all clusters in the clustering model with the structural feature vector of the web page.

5. The method according to claim 1, characterized in that, Based on the respective implementation structures of the two web pages, determine whether the two web pages are related pages, including: If the two web pages have similar implementation structures, they are determined to be closely related pages. The steps for obtaining the implementation structure are as follows: input the structural feature vector of the web page into the corresponding clustering model to obtain the cluster to which the web page belongs. Each cluster corresponds to a main structure of the web page, that is, the main structures of web pages in different clusters are also different. The implementation structure of the web page is obtained by combining the function and structure formed by the cluster to which the web page belongs with the functional tags of the web page.

6. The method according to claim 1, characterized in that, Based on the respective functions and structures of the two web pages, the method for determining whether the two web pages are related pages further includes: Obtain the semantic theme of each of the two web pages, where the semantic theme represents the content theme described by the web pages; If the two web pages have similar functions and structures, and the two web pages have the same semantic theme, then the two web pages are determined to be closely related pages.

7. A device for identifying closely related web pages based on function and structure, characterized in that, The device includes: The information acquisition module is used to acquire page information of the two web pages to be identified; The function recognition module is used to extract function features from the two web pages, obtain function feature vectors for the two web pages respectively, and input the function feature vectors into the function classification model to predict the function labels of the two web pages. The structural feature extraction module is used to find the corresponding clustering model in the clustering model set according to the functional label, extract the structural features of the two web pages, and obtain the structural feature vectors of the two web pages respectively. The function and structure generation module is used to input the structural feature vector into the corresponding clustering model to predict the cluster to which the two web pages belong. The cluster to which the web page belongs and the function tag of the web page constitute the structure and function of the web page. The judgment module is used to determine whether the two web pages are related pages based on their respective functions and structures.

8. The apparatus according to claim 7, characterized in that, The function recognition module includes: The first labeling module is used to label the training data according to a predefined set of functional categories to obtain a set of labeled web pages for model training; The first feature processing module is used to perform feature engineering processing on the tagged Web page set to obtain a functional feature matrix for each Web page in the tagged Web page set, wherein the features in the functional feature matrix include at least the number of tags, text information, manual rules and semantic information; The classification training module is used to train the functional classification model based on the feature matrix of each Web page to obtain the functional classification model; The incremental learning module is used to correct labels when predicting the function of a web page if the function label is incorrect, so that the function classification model can perform incremental learning on the corrected data.

9. The apparatus according to claim 7, characterized in that, The function and structure generation module includes: The second tagging module is used to divide the tagged Web page set into multiple Web page sets according to functional tags, and the Web pages in each Web page set have the same functional tags; The second feature processing module is used to perform feature engineering on each web page in each web page set to obtain a structural feature matrix for each web page, and to train a clustering model using the structural feature matrix to obtain a set of clustering models corresponding to multiple functional labels.

10. An electronic device, characterized in that, It includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor, when executed, implements the close relative web page identification method based on function and structure as described in any one of claims 1-6.

Images