Webpage data collection method, electronic device, and computer-readable storage medium

Abstract

The application provides a webpage data collection method, an electronic device and a computer readable storage medium. The method comprises the following steps: obtaining original HTML text of a reference webpage; preprocessing the original HTML text to obtain target HTML text corresponding to the reference webpage, wherein the target HTML text comprises a plurality of attribute tags related to data collection; generating a large model according to the target HTML text and a rule obtained by pre-training, generating data collection rule information, wherein the data collection rule information comprises at least one content collection rule item and at least one page relationship rule item, each content collection rule item is used for describing a collection rule of a content element in a webpage, and each page relationship rule item is used for describing a jump rule between webpages; and collecting data of the webpages in the network based on the data collection rule information. The method has strong universality and reusability, and can realize automatic data collection of the whole link.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to a webpage data collection method, an electronic device and a computer readable storage medium. BACKGROUND

[0002] With the continuous development of Internet technology, a large amount of information resources has been accumulated in the Internet, which has important value for scientific research, business and many other fields. Therefore, how to efficiently and accurately collect the required data from the Internet is a problem worth studying.

[0003] With the rapid development of large models, the existing technology proposes to collect Internet data based on large models. Specifically, the address of the webpage is obtained to obtain the webpage code, and then the webpage code and the target template are used to generate prompt words, which are input into the large model for processing to obtain the collection result.

[0004] However, the reusability of the method of the prior art is weak, and it has strong dependence on the large model, resulting in low data collection efficiency. In addition, the method of the prior art may not meet the timeliness requirement and accuracy requirement when applied in the financial field. SUMMARY

[0005] The purpose of the present application is to provide a webpage data collection method, an electronic device and a computer readable storage medium to solve the problems of weak reusability, low collection efficiency and difficulty in meeting the timeliness requirement and accuracy requirement in the financial field in the prior art.

[0006] To achieve the above purpose, the technical solutions adopted by the embodiments of the present application are as follows:

[0007] In a first aspect, the present application provides a webpage data collection method, comprising:

[0008] obtaining the original HyperText Markup Language (HTML) text of a reference webpage;

[0009] preprocessing the original HTML text to obtain the target HTML text corresponding to the reference webpage, wherein the target HTML text includes a plurality of attribute tags related to data collection;

[0010] generating a large model according to the target HTML text and a pre-trained rule to generate data collection rule information, wherein the data collection rule information includes at least one content collection rule item and at least one page relationship rule item, each content collection rule item is used to describe the collection rule of a content element in a webpage, and each page relationship rule item is used to describe the jump rule between webpages;

[0011] Collect data of a webpage in the network based on the data collection rule information.

[0012] In a second aspect, the present application provides a webpage data collection device, comprising:

[0013] An obtaining module, configured to obtain original HyperText Markup Language (HTML) text of a reference webpage;

[0014] A preprocessing module, configured to preprocess the original HTML text to obtain target HTML text corresponding to the reference webpage, wherein the target HTML text comprises a plurality of attribute tags related to data collection;

[0015] A generating module, configured to generate a large model according to the target HTML text and a rule obtained by pre-training, and generate data collection rule information, wherein the data collection rule information comprises at least one content collection rule item and at least one page relationship rule item, each content collection rule item is used to describe a collection rule of a content element in a webpage, and each page relationship rule item is used to describe a jump rule between webpages;

[0016] A collecting module, configured to collect data of a webpage in the network based on the data collection rule information.

[0017] In a third aspect, the present application provides an electronic device, comprising a processor and a memory, wherein the memory stores machine readable instructions executable by the processor, and when the electronic device is running, the processor executes the machine readable instructions to perform the steps of the webpage data collection method in the first aspect.

[0018] In a fourth aspect, the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is run by a processor, the steps of the webpage data collection method in the first aspect are executed.

[0019] The webpage data collection method, the electronic device and the computer readable storage medium provided by the application first simplify the original HTML text of the reference webpage into target HTML text related to the data collection scene, and then input the target HTML text into a large model to obtain data collection rule information with universality and reusability, so that data collection can be performed based on the data collection rule information in the subsequent data collection process. Through the above process, the data collection rule information has great universality and reusability, and therefore can be applied to data collection in various scenes. In addition, the general collection rule is directly used for data collection in the data collection process, without relying on the large model. At the same time, since the data collection is performed using the collection rule, the collection rule has the characteristics of simplicity and light weight, and therefore, efficient, fast and accurate data collection can be realized, meeting the scenes of financial and other industries that require high real-time and accuracy of data. Further, since the data collection rule information generated by the application contains various content collection rule items and various page relationship rule items, on the one hand, the content collection rule items can be used to collect the content of the webpage itself, and on the other hand, the page relationship rule items can be used to obtain the content of other pages related to the page, so as to collect complete and multi-level webpage content and realize automatic data collection of the whole link. BRIEF DESCRIPTION OF DRAWINGS

[0020] In order to more clearly illustrate the technical solutions of the embodiments of the application, the following will briefly introduce the drawings needed to be used in the embodiments. It should be understood that the following drawings only show some embodiments of the application, and therefore should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can also be obtained without creative labor.

[0021] Figure 1 An exemplary scene diagram for the embodiments of the application;

[0022] Figure 2 A flowchart of the webpage data collection method provided by the embodiments of the application;

[0023] Figure 3 An example of a list page containing list links and next page links;

[0024] Figure 4 An example of a detail page containing title, author and body;

[0025] Figure 5 A rule generation flowchart of the webpage data collection method provided by the embodiments of the application;

[0026] Figure 6 A rule generation flowchart of the webpage data collection method provided by the embodiments of the application for list pages;

[0027] Figure 7 A rule information optimization and data collection flowchart of the web page data collection method provided by the embodiment of the present application;

[0028] Figure 8 A flowchart of data collection according to target data collection rule information of the web page data collection method provided by the embodiment of the present application;

[0029] Figure 9 An example diagram of the data collected by the embodiment of the present application;

[0030] Figure 10 A preprocessing flowchart of the web page data collection method provided by the embodiment of the present application;

[0031] Figure 11 A complete flowchart of the web page data collection method provided by the embodiment of the present application;

[0032] Figure 12 A module structure diagram of the web page data collection device provided by the embodiment of the present application;

[0033] Figure 13 A structure schematic diagram of the electronic device 130 provided by the embodiment of the present application. DETAILED DESCRIPTION

[0034] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. It should be understood that the drawings in the present application only serve the purpose of description and illustration, and are not used to limit the protection scope of the present application. In addition, it should be understood that the schematic drawings are not drawn according to the actual proportions. The flowcharts used in the present application show the operations implemented according to some embodiments of the present application. It should be understood that the operations of the flowcharts can not be implemented in sequence, and the steps without logical contextual relationship can be reversed in sequence or implemented simultaneously. In addition, one or more other operations can be added to the flowcharts or one or more operations can be removed from the flowcharts under the guidance of the content of the present application by those skilled in the art.

[0035] In addition, the described embodiments are only some of the embodiments of the present application, not all the embodiments. The components of the embodiments of the present application described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the claimed present application, but only represents selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0036] It should be noted that the term "comprising" will be used in the embodiments of the present application to indicate the presence of the features that follow, but not to exclude the addition of other features.

[0037] Although the prior art provides a method of collecting web page data using a large model, this method generates a prompt word by combining the web page code and the target template after obtaining the web page code, and inputs the prompt word into the large model for processing to obtain the collection result. As can be seen, this method directly executes data collection by the large model and outputs the collection result. This way of directly executing data collection by the large model has the following problems:

[0038] First, even for some data collection tasks with high similarity, a large model needs to be called to complete data collection, which results in weak reusability of data collection.

[0039] Second, all data collection tasks need to be completed by a large model, resulting in strong dependence on the large model.

[0040] Third, in the financial field, there are massive amounts of data every day, so the amount of data that needs to be collected every day is huge, and the real-time requirement for data acquisition is high in the financial field. If a large model is directly used for data collection, the real-time requirement may not be met.

[0041] Fourth, the financial field has high requirements for the accuracy of data, and since the current large model still has certain limitations in processing complex structured long text documents, there may be a problem that it is difficult to guarantee the accuracy of the collection result when data collection is completed by a large model.

[0042] In view of the above problems, the embodiments of the present application provide a web page data collection method, which no longer relies on a large model to complete data collection, but only analyzes and extracts collection rules of various page elements in the web page by a large model. These rules are general rules. On this basis, data collection is performed based on these general collection rules, so as to meet the requirements of reusability, accuracy and real-time without relying on a large model.

[0043] Before introducing the technical solutions of the embodiments of the present application, it needs to be first explained that the web page data collection method provided by the embodiments of the present application must obtain the user's consent in an explicit form before collecting and generating personal information and important data in the execution process, and can be applied to the solutions of the embodiments of the present application. At the same time, the web page data collection method provided by the embodiments of the present application must follow the principles of legality, legitimacy and necessity, and must follow the rules of public collection and use, and must explicitly show the purpose, method and scope of collecting and using information. The related data in the above examples do not include personal information irrelevant to the related services provided by the above examples.

[0044] Figure 1 As shown in an exemplary scenario diagram of the embodiments of the present application, Figure 1 the embodiments of the present application can be applied to a data collection scenario in the financial field. In this data collection scenario, a certain financial analysis platform needs to initiate and execute a data collection task at one or more fixed times every day. When the data collection task is executed, the data collection task collects data from the Internet through the collection rule information obtained in advance based on a large model, and further data analysis is performed by the financial analysis platform.

[0045] It is worth noting that the scenario shown in the above Figure 1 is only an example. Since the rule information collected by the embodiments of the present application is universal, the embodiments of the present application can be applied to any scenario that needs to collect web page data.

[0046] Figure 2 The flowchart of the web page data collection method provided by the embodiments of the present application, which can be applied to any electronic device with computing processing capability. As shown in Figure 2 , the method comprises:

[0047] S201, obtaining the original Hyper Text Markup Language (HTML for short) text of a reference web page.

[0048] Optionally, the reference web page can be any web page on the Internet. The reference web page can include various elements. For example, when the reference web page is a list page, the reference web page can include elements such as title, list link, next page link, etc. When the reference web page is a detail page, the reference web page can include elements such as title, author, body, etc. Figure 3 For an example of a list page containing list links and next page links, Figure 4 for an example of a detail page containing title, author, body.

[0049] Optionally, various elements of the reference webpage can be represented in the form of HTML text. Specifically, each element in the reference webpage can be recorded in an attribute tag in the HTML text. For example, the title in the reference webpage can be recorded in the "title" tag in the HTML text. By reading and parsing the "title" tag, the title of the reference webpage can be obtained.

[0050] In a specific implementation process, a user can flexibly select a webpage in the Internet as the reference webpage. For example, the user can select a list page containing both list links and next page links as the reference webpage, so that more complete rule information can be quickly extracted.

[0051] As an example, the user can input the address of the reference webpage, and the electronic device can download the original HTML text of the reference webpage from the address based on the Playwright dynamic rendering method. Playwright can capture dynamic content by actually running the JavaScript code in the webpage, thereby obtaining complete HTML text. It should be understood that the original HTML refers to the directly downloaded and unmodified HTML text.

[0052] Optionally, the webpage data collection process of the embodiments of the present application can be represented by a six-tuple model S=(U,R,D,P,V,E), where U represents the webpage address request submitted by the user; R represents dynamically loading webpage content through Playwright; D represents preprocessing of the original HTML text; P represents generating data collection rule information based on a large language model; V represents effectiveness evaluation and correction of the generated rules; E represents manual verification and distributed crawler scheduling mechanism.

[0053] Optionally, the use process of R in the above six-tuple model can be represented as H=R(u ), u∈U, where u represents the address of the reference webpage input by the user; H represents the original HTML text obtained after dynamic rendering by Playwright technology; Playwright simulates browser behavior, runs JavaScript code, and ensures that complete DOM structure and asynchronous loading content are obtained.

[0054] S202, preprocessing the original HTML text to obtain a target HTML text corresponding to the reference webpage, the target HTML text including a plurality of attribute tags related to data collection.

[0055] Optionally, a complete original HTML text can include a large number of attribute tags, which include both content-related tags and format-related tags. In the data collection scenario involved in the embodiments of the present application, all attribute tags are not required to be used. In view of this problem, in this step, the original HTML text of the reference webpage can be preprocessed to remove the attribute tags irrelevant to the data collection scenario from the original HTML text, and only the attribute tags related to the data collection scenario are retained, so as to simplify the structure of the HTML text and improve the processing efficiency when generating the collection rule subsequently.

[0056] The preprocessing of the original HTML text can be represented as H’=D(H), specifically, each node n∈H in the DOM tree of the original HTML text is traversed; for the attribute set of each node, only the attributes belonging to the key attribute set {id, class, data-*, …} are retained; all attributes not belonging to the key attribute set are removed; and the DOM tree of the original HTML text is reconstructed based on the retained attributes. <script>和<style>标签及其内容；输出简化后的目标HTML文本H’，供后续规则生成使用。

[0057] S203、根据上述目标HTML文本以及预先训练得到的规则生成大模型，生成数据采集规则信息。该数据采集规则信息中包括至少一个内容采集规则项以及至少一个页面关系规则项，各内容采集规则项分别用于描述网页中一项内容的采集规则，各页面关系规则项分别用于描述网页之间的跳转规则。

[0058] 可选地，上述大模型可以预先通过训练得到，大模型具有分析输出网页中各项元素的采集规则的能力。

[0059] 可选地，上述数据采集规则信息中包括至少一个内容采集规则项以及至少一个页面关系规则项，其中，各内容采集规则项分别用于描述网页中一项内容元素的采集规则，各页面关系规则项分别用于描述网页之间的跳转规则。也即，内容采集规则项是与网页中的具体内容相关的采集规则项，页面关系规则项是与网页之间跳转相关的采集规则项。

[0060] 可选地，上述的网页中的内容元素可以是指链接以外的元素。示例性地，内容元素可以包括：标题、作者、发布时间、正文、来源等。相应地，上述内容采集规则项可以包括：标题的采集规则项、作者的采集规则项、发布时间的采集规则项、正文的采集规则项以及来源的采集规则项等。

[0061] 可选地，上述页面关系规则项可以包括：列表链接的采集规则项、下一页链接的采集规则项以及附件链接的采集规则项等。

[0062] 可选地，上述各内容采集规则项可以用于描述网页中一项内容元素的采集规则，在执行数据采集时，该内容采集规则项可以被执行，从而获取到对应的内容。上述各页面关系规则项可以用于描述网页之间的跳转规则，在执行数据采集时，该页面关系规则项可以被执行，从而获取到即将跳转到的页面的地址，并据此获取跳转到的页面的内容。

[0063] 作为一种示例，上述各内容采集规则项以及页面关系采集规则项可以通过XPath或JPath等格式来表示。以通过XPath表示为例，将参考网页的目标HTML文本输入大模型后，大模型可以输出各内容元素以及列表链接等跳转元素对应的XPath表达式。该XPath表达式被执行后即可获取到对应的内容。

[0064] 值得说明的是，由于上述参考网页可以是任选的网页，将其HTML文本输入经过训练的大模型后，大模型输出内容采集规则项以及页面关系规则项，在此过程中，无需受到某种场景的限制，因此，所得到的各采集规则项具有极高的重用性，可以应用于后续的任何数据采集过程中。

[0065] 上述生成采集规则信息的过程可以表示为(G,Rc,Rp)=P(H’)，其中：G为页面结构图，表示为图结构 G=(V,E)，其中 V 表示内容区块节点，E 表示区块间的关系 Rc:内容采集规则集合，形式为Rc={ri∣ri=(name,type,selector)}，其中name表示字段名称(如标题、正文)，type表示字段类型，selector表示xpath或jpath表达式。Rp：页面导航关系规则集合，用于定义子页、下一页等跳转路径，形式为Rp={pj∣pj=(relation,selector)}，其中relation表示页面关系类型(如"子页","下一页")，selector表示对应的xpath表达式。

[0066] S204、基于上述数据采集规则信息，对网络中的网页进行数据采集。

[0067] 经过前述步骤S201-S203，即可生成具有通用性和重用性的数据采集规则信息，进而可以根据实际需要，利用上述数据采集规则信息进行数据采集。

[0068] 作为一种可选的方式，用户可以配置采集任务，如设置该数据源的任务名称，采集的优先级，推送的具体项目及定时调度的策略等。配置完成之后点击提交则会自动生成定时任务。进一步地，如果时间到达定时任务配置的时间节点则开始启动分布式爬虫调度，以消息队列为中间件进行分布式调度和运行爬虫。进一步的，分布式爬虫调度分为主节点模式和工作节点模式，定时任务发送到主节点，主节点根据配置的优先级及先后顺序将采集任务分发到空闲的工作节点，工作节点则负责执行具体的业务采集，可表示为：M= { f_i(c_i, p_i) | ∀c_i ∈ C, p_i ∈ P}，其中：f_i表示针对第i个规则配置的处理函数，c_i表示第i层级的采集配置规则，p_i表示网页中对应的内容片段，整体表示对每层规则和对应网页内容片段的映射处理，采集结果M是通过将多层级的采集配置规则C与网页内容P进行规则化处理得到的结构化数据，其中配置规则C包含任务优先级、字段提取规则、数据转换规则等参数，网页内容P是通过Playwright动态下载器获取的原始网页数据。

[0069] 本实施例中，首先将参考网页的原始HTML文本精简为与数据采集场景相关的目标HTML文本，再将目标HTML文本输入大模型中得到具有通用性以及重用性的数据采集规则信息，从而可以在后续数据采集过程中，基于该数据采集规则信息进行数据采集。经过上述过程，使得数据采集规则信息具有极大的通用性以及重用性，因此可以应用于各种场景的数据采集。另外，在数据采集过程中直接使用通用的采集规则进行数据采集，无需依赖大模型。同时，由于是使用采集规则进行数据采集，采集规则具有简洁、轻量化的特点，因此，可以实现高效快速准确的数据采集，满足金融等对数据实时性和准确性要求较高的场景。进一步地，由于本申请所生成的数据采集规则信息中既包含了各种内容采集规则项，又包含了各种页面关系规则项，因此，一方面，可以利用内容采集规则项采集到网页本身的内容，另一方面，可以利用页面关系规则项获取到与页面相关的其他页面的内容，从而采集得到完整的、多层级的网页内容，实现全链路的自动化数据采集。

[0070] 上述步骤S203的一种可选方式为：选择包含所有类型的页面元素的网页作为参考网页，根据该参考网页的HTML文本生成提示词，将提示词输入上述大模型中，即可得到包含各种所需的内容采集规则项以及页面关系规则项的数据采集规则信息。

[0071] 以下介绍上述步骤S203的另一种可选方式。

[0072] 图5为本申请实施例提供的网页数据采集方法的规则生成流程图，如图5所示，上述步骤S203的另一种可选方式包括：

[0073] S501、根据上述目标HTML文本，生成第一提示词。

[0074] 可选地，可以由用户基于上述目标HTML文本手动生成提示词，例如用户可以在电子设备中的界面中输入"请以下述HTML文本为基础生成数据采集规则”这一提示文本，将该提示文本以及上述目标HTML文本共同作为上述第一提示词输入上述大模型中。或者，还可以由脚本程序自动生成包含上述第一目标HTML文本的第一提示词。

[0075] S502、将上述第一提示词输入上述规则生成大模型，由上述规则生成大模型输出上述参考网页对应的多个第一参考规则项。

[0076] 其中，各第一参考规则项分别为前述的一个内容采集规则项或一个页面关系规则项。

[0077] 对于不同的参考网页，大模型输出的第一参考规则项可能存在不同。例如，某个参考网页中仅包含列表链接和下一页链接，相应地，大模型输出的第一参考规则项仅包含页面关系规则项。又例如，某个参考网页中既包含列表链接和下一页链接，又包含标题和作者等内容元素。相应地，大模型输出的第一参考规则项既包含页面关系规则项，又包含部分或全部的内容采集规则项。又例如，某个参考网页中仅包含标题和作者等内容元素。相应地，大模型输出的第一参考规则项仅包含内容采集规则项。

[0078] S503、根据上述参考网页的类型以及上述多个第一参考规则项，生成上述数据采集规则信息。

[0079] 如上所述的，用户所选择的参考网页不同，由大模型所输出的第一参考规则项可能不同。基于此，可以根据参考网页的类型，对第一参考规则项进行相应的处理，从而得到上述数据采集规则信息。

[0080] 本实施例中，对于不同类型的参考网页，大模型输出的第一参考规则项不同，因此，可以基于参考网页的类型以及大模型输出的第一参考规则项进行对应的处理，以保证得到完整的内容采集规则项以及页面关系规则项的。在此过程中，无需用户按照特定要求选择参考网页，即用户可以选择任何的网页作为参考网页，也无需用户执行任何额外的操作即可得到数据采集规则信息。

[0081] 作为一种可选的实施方式，本申请实施例中将网页划分为两种类型，一种为列表页，另一种为详情页。其中，在列表页中包括列表链接以及下一页等跳转元素，同时，列表页中可能不包含内容元素，或者，包含部分内容元素。一个示例中，列表页中除了包含列表链接和下一页链接外，还包含标题和发布日期。另一个示例中，列表页中仅包含列表链接和下一页链接。

[0082] 基于上述的网页类型划分策略，上述步骤S303可以包含多种可能的实现方式。

[0083] 一种可选的实现方式中，如果上述参考网页为详情页，则可以在得到上述多个第一参考规则项之后，自动从网络中搜索包含列表链接以及下一页链接的网页，并自动根据该网页的HTML文本生成提示词并输入大模型，从而得到多个页面关系规则项。将这些页面关系规则项与上述多个第一参规则项结合，可以得到完整的数据采集规则信息。

[0084] 另一种可选的实现方式中，如果上述参考网页为列表页，则可以按照下述过程生成完整的数据采集规则信息。

[0085] 图6为本申请实施例提供的网页数据采集方法的针对列表页的规则生成流程图，如图6所示，上述步骤S503的另一种可选方式包括：

[0086] S601、若参考网页为列表页，则获取参考网页中至少一个列表链接所链接到的详情页的HTML文本。

[0087] 示例性地，参考网页中包含如上述图3所示例的3个列表链接，则可以选择这3个列表链接中的一个或多个，分别获取所选择的列表链接所链接到的详情页的HTML文本。

[0088] 可选地，大模型在输出上述多个第一参考规则项之后，可以执行其中的列表链接的采集规则项，具体可以是执行列表链接对应的XPath表达式，该XPath表达式执行后，可以采集到参考网页中各列表链接所链接到的详情页的HTML文本。进而可以选择其中的至少一个HTML文本。

[0089] S602、根据各详情页的HTML文本，生成各详情页对应的第二提示词。

[0090] S603、将上述第二提示词输入上述规则生成大模型，由上述规则生成大模型输出各详情页对应的多个第二参考规则项，各第二参考规则项分别为一个内容采集规则项。

[0091] 可选地，由于输入的第二提示词为详情页对应的提示词，因此，大模型据此输出的第二参考规则项为内容采集规则项。

[0092] S604、根据上述多个第一参考规则项以及各详情页对应的多个第二参考规则项，生成数据采集规则信息。

[0093] 上述第一参考规则项为列表页对应的规则项，上述各第二参考规则项为各详情页对应的规则项，通过对第一参考规则项以及各第二参考规则项进行比较分析，可以选择出效果最优的规则项，从而保证数据采集规则信息的准确性。

[0094] 本实施例中，利用了列表页可以链接到其他详情页这一特点，可以获取列表页所链接到的至少一个详情页的规则信息，进而可以根据这些规则信息以及列表页原本的第一参考规则，分析选择出效果最优的规则项，从而保证数据采集规则信息的准确性。

[0095] 作为一种可选的实施方式，上述步骤S604包括：

[0096] 分别测试上述多个第一参考规则项中的各内容采集规则以及各详情页对应的各第二参考规则项的信息抽取结果；根据上述多个第一参考规则项中的各内容采集规则项以及各详情页对应的各第二参考规则项的信息抽取结果，从上述多个第一参考规则项中的各内容采集规则项以及各详情页对应的各第二参考规则项中选择出所述数据采集规则信息中的各内容采集规则项，并将上述多个第一参考规则项中的各页面关系规则项分别作为所述数据采集规则信息中的一个页面关系规则项。

[0097] 可选地，可以分别执行第一参考规则项中各内容采集规则项的XPath表达式以及各第二参考规则项的XPath表达式，从而得到第一参考规则项中各内容采集规则项的信息抽取结果以及各第二参考规则项的信息抽取结果。进而，可以对信息抽取结果进行比对，将抽取结果最优的规则项作为数据采集规则信息中的内容采集规则项。另外，将第一参考规则项中原本的页面关系规则项直接作为数据采集规则信息中的页面关系规则项。

[0098] 通过上述方式，基于实际的抽取结果进行比对，可以从第一参考规则项和第二参考规则项中选择出效果最优的规则项，保证数据采集规则信息的准确性。

[0099] 作为另一种可选方式，如果第二参考规则项中不存在内容采集规则项，则可以对各第二参考规则项进行信息抽取结果比对，并从各第二参考规则项中选择信息抽取结果最优的内容采集规则项作为数据采集规则信息中的内容采集规则项。

[0100] 值得说明的是，在具体实施过程中，需要对各种内容采集规则项分别执行上述操作。示例性地，内容采集规则项包括：标题采集规则项和正文采集规则项。对于标题采集规则项，需要获取第一参考规则项中标题采集规则项的信息抽取结果以及各详情页对应的第二参考规则项中标题采集规则项对应的信息抽取结果，并进行比对。示例性地，假设第一参考规则项中的标题采集规则项为标题规则项1，且获取到了参考网页所链接到的3个详情页的第二参考规则项，假设3个第二参考规则项中的标题采集规则项分别为标题规则项2、标题规则项3、标题规则项4，则可以对标题规则项1、标题规则项2、标题规则项3以及标题规则项4的信息抽取结果进行比对，从而选择出这4个规则项中最优的一个标题规则项作为数据采集规则信息中的标题规则项。对于正文采集规则项，也需要按照上述过程以确定出数据采集规则信息中的正文规则项，此处不再赘述。

[0101] 以下对前述步骤S204的实施方式进行说明。

[0102] 一种可能的方式中，可以直接利用步骤S203所得到的数据采集规则信息对网络中的网页进行数据采集。具体地，由前述实施例中所述的工作节点按照数据采集规则信息中的各规则项执行数据采集操作。

[0103] 另一种可能的方式中，可以首先对数据采集规则信息进行优化，得到效果更优的数据采集规则信息，并据此进行数据采集。以下对这种方式进行详细说明。

[0104] 图7为本申请实施例提供的网页数据采集方法的规则信息优化及数据采集流程图，如图7所示，上述步骤S204具体可以包括：

[0105] S701、基于数据采集规则信息对具有相同布局的多个验证网页进行数据采集，得到各验证网页的数据采集结果。

[0106] 示例性地，上述具有相同布局的多个验证网页例如可以是包含列表链接和下一页链接，且列表链接的链接内容、位置、数量相同，且下一页链接的位置的链接内容相同的多个网页。针对这些验证网页，分别使用上述的数据采集规则信息对这些验证网页进行数据采集，可以基于采集结果衡量出数据采集规则信息的通用性的优劣。

[0107] S702、获取具有相同布局的多个验证网页的HTML文本。

[0108] S703、将各验证网页的数据采集结果以及验证网页的HTML文本分别作为一组输入数据输入至预先训练得到的语义分析大模型中，得到分析结果。

[0109] 示例性地，假设验证网页的数量为3个，分别为网页1、网页2和网页3，可以将网页1的数据采集结果和HTML文本作为一组输入数据，将网页2的数据采集结果和HTML文本作为一组输入数据，将网页3的数据采集结果和HTML文本作为一组输入数据。这三组输入数据输入大模型后，大模型对每组输入数据内的数据采集结果和HTML文本从语义层面进行分析，以判断数据采集结果的完整性和准确性。同时，大模型对三组输入数据中的数据采集结果统一进行语义分析，以判断数据采集规则信息在相同布局的网页上的通用性。

[0110] S704、根据上述分析结果，对上述数据采集规则信息进行优化处理，得到目标数据采集规则信息。

[0111] 可选地，大模型在判断出数据采集结果的完整性、准确性和通用性之后，可以输出相应的修改建议。电子设备可以根据修改建议相应对数据采集规则信息进行优化处理，例如调整其中部分规则项的规则等，从而得到优化后的数据采集规则信息，进而可以由人工确认是否使用优化后的数据规则信息，若是，则将其作为上述目标数据采集规则信息。

[0112] S705、基于上述目标数据采集规则信息，对网络中的网页进行数据采集。

[0113] 本实施例中，选择具有相同布局的多个验证网页进行数据采集，由大模型根据实际的采集结果以及验证网页自身的HTML文本，分析出数据采集规则信息在完整性和准确性维度上的优劣，并据此对数据采集规则信息进行优化。由于选择的是具有相同布局的多个验证网页，因此，可以衡量出数据采集规则信息在通用性上的优劣，相应地，在优化时，可以从通用性上对数据采集规则信息进行优化，从而使得数据采集规则信息在通用性、完整性和准确性维度上均得以进一步提升。

[0114] 图8为本申请实施例提供的网页数据采集方法的根据目标数据采集规则信息进行数据采集的流程图，如图8所示，上述步骤S705可以包括：

[0115] S801、获取当前待提取网页的HTML文本。

[0116] 由前述的工作节点执行网页HTML文本获取以及下述的数据采集。工作节点在执行数据采集时，解析目标数据采集规则信息中的各规则项，将各规则项传递给字段解析器，并将采集参数传递给Playwright动态下载器。其中，采集参数例如可以包括网页地址、推动到业务队列的字段以及任务采集优先级等信息。Playwright根据采集参数对当前待提取网页进行访问并获取其HTML文本并传递给字段解析器。

[0117] S802、执行上述目标数据采集规则信息中的各内容采集规则项以及各页面关系规则项，以对当前待提取网页的HTML进行数据采集。

[0118] 可选地，字段解析器获取到待提取网页的HTML文本后，执行各采集项，以完成数据采集提取。

[0119] S803、将各内容采集规则项执行后所得到的第一采集结果进行保存。

[0120] 对于各内容采集规则项，其执行之后可以直接得到页面内容元素，例如标题、发布时间等，因此，可以将其进行保存。

[0121] S804、根据各页面关系规则项执行后所得到的第二采集结果，依次对与所述当前待提取网页存在跳转关系的各网页进行数据采集。

[0122] 对于各页面关系规则项，其执行之后得到的并非是最终的页面内容元素，而是关于页面的跳转信息，因此，需要根据不同的页面关系规则项分别进行处理。

[0123] 以下具体进行说明。

[0124] 页面关系规则项可以包括列表链接规则项以及下一页规则项，另外，还可以包括附件链接规则项。附件链接规则项用于获取一个页面中的附件的链接，因此，附件链接规则项的处理方式与列表链接规则项的处理方式相同，可以参考下述对于列表链接规则项的处理方式，不再单独赘述。

[0125] 对于列表链接规则项来说，若列表链接规则项执行后所得到的第二采集结果不为空，则按照第二采集结果中的各网页链接的顺序，依次向各网页链接所指示的网页发送访问请求，以获取各网页链接所指示的网页的网页内容。

[0126] 列表链接规则项之后，所得到的第二采集结果为所链接到的网页链接，也即网页地址。

[0127] 一种可能的方式中，第二采集结果中的各网页链接所链接到的网页为详情页，例如包含正文内容，则可以获取所链接到的网页的网页内容并进行存储。另一种可能的方式中，第二采集结果中的各网页链接所链接到的网页仍然为列表网页，则需要进一步解析列表网页中的列表链接并进一步按照该列表链接进行网页访问，不断递归进行，直至链接到的网页为详情页，将其内容进行保存。

[0128] 对于下一页规则项来说，若下一页规则项执行后所得到的第二采集结果不为空，则获取第二采集结果中下一页链接的目标位置，并通过动态下载器执行点击该目标位置的操作，以获取下一页链接所指向的网页的网页链接，并向下一页链接所指向的网页发送访问请求，以获取下一页链接所指向的网页的网页内容。

[0129] 下一页规则项在执行之后，所得到的第二采集结果为下一页链接在当前待提取网页中的位置，即上述目标位置，因此，基于该第二采集结果，无法直接访问网页。因此，在本实施例中，通过动态下载器执行点击该目标位置的操作，以获取下一页链接所指向的网页的网页链接，并向下一页链接所指向的网页发送访问请求，从而获取并存储其网页内容。

[0130] 在上述各规则项执行之后保存第一采集结果和第二采集结果时，还可以通过如下的过程实现去重。

[0131] 如果当前待提取网页不为第一页，本页字段解析结果与上一页的字段解析结果进行对比，重复则认为本页与上页重复，结束爬虫采集。在完成数据后，上传网页HTML和正文内容HTML到对象存储服务（Object Storage Service，简称OSS）文件对象存储，获取文件的MD5值。如果正文内容包含图片等资源信息则将对应的图片也上传到OSS文件对象存储中，并将对应的MD5写回到正文内容的指定位置。这种方式保证了正文内容的所有资源在下载后都直接使用，避免资源相对路径引用丢失。另外，分别计算MD5与网页地址的拼接结果和字段解析结果的MD5值作为页面标识符，在redis缓存当中搜索，若存在则认为本页已进行数据采集，结束爬虫采集；若不重复，则将页面标识符加入redis，以避免数据的重复写入。进一步地，将字段解析结果转换为与数据库对应的数据模型对象，将数据模型对象装载入流水线缓存，流水线缓存会在数据模型对象积累到一定数量或一定的间隔时间后，将数据模型对象批量入库数据库，避免对数据库的频繁写入；在完成数据入库后，将对应解析结果MD5值写入到redis进行后续去重，并通过消息队列推送到下游进行消费处理。

[0132] 图9为通过本申请实施例所采集到的数据的示例图，参照图9，经过本申请实施例的数据采集过程，可以采集到完整的多嵌套层次的网页内容。其中，图9中所示的子页面，可以包括列表链接所链接到的网页或者下一页链接所链接到的网页。

[0133] 图10为本申请实施例提供的网页数据采集方法的预处理流程图，如图10所示，上述步骤S202可以包括：

[0134] S1001、根据原始HTML文本，确定参考网页的树形结构。

[0135] 参考网页的树形结构可以是通过文档对象模型（Document Object Model，简称DOM）解析得到的DOM树形结构。

[0136] S1002、基于递归上下文策略对树形结构进行元素筛选，得到待保留元素集合。

[0137] 对树形结构中的各元素，递归添加其祖先节点、后代节点以及同级节点，并在每次递归迭代中减小深度阈值、子节点数阈值以及同级节点数量阈值，从而逐步缩小范围，使得关键节点周围的次要信息被选择性保留，得到上述待保留元素集合。

[0138] S1003、根据待保留元素集合，逆序遍历树形结构，以从树形结构中删除至少一个元素，得到目标树形结构。

[0139] 基于待保留元素集合，逆序遍历树形结构，从树形结构中删除以下三类节点：不在待保留元素集合中的节点、不包含文本内容或属性的节点、子节点数量小于或等于1且属于非根节点的节点。

[0140] S1004、基于目标树形结构，生成目标HTML文本。

[0141] 本实施例中，通过递归上下文策略以及逆序遍历策略，使得原始HTML文本中与数据采集相关的信息被保留，与数据采集无关的信息被剔除，从而提升后续规则提取和数据采集的效率。

[0142] 图11为本申请实施例提供的网页数据采集方法的完整流程图，其中各步骤的具体过程已在前述各实施例中描述，不再赘述。

[0143] 基于同一发明构思，本申请实施例中还提供了与网页数据采集方法对应的网页数据采集装置，由于本申请实施例中的装置解决问题的原理与本申请实施例上述网页数据采集方法相似，因此装置的实施可以参见方法的实施，重复之处不再赘述。

[0144] 图12为本申请实施例提供的网页数据采集装置的模块结构图，如图12所示，该装置包括：

[0145] 获取模块1201，用于获取参考网页的原始超文本标记语言HTML文本。

[0146] 预处理模块1202，用于对所述原始HTML文本进行预处理，得到所述参考网页对应的目标HTML文本，所述目标HTML文本中包括与数据采集相关的多个属性标签。

[0147] 生成模块1203，用于根据所述目标HTML文本以及预先训练得到的规则生成大模型，生成数据采集规则信息，所述数据采集规则信息中包括至少一个内容采集规则项以及至少一个页面关系规则项，各所述内容采集规则项分别用于描述网页中一项内容元素的采集规则，各所述页面关系规则项分别用于描述网页之间的跳转规则。

[0148] 采集模块1204，用于基于所述数据采集规则信息，对网络中的网页进行数据采集。

[0149] 作为一种可能的实施方式，所述生成模块1203具体用于：

[0150] 根据所述目标HTML文本，生成第一提示词；

[0151] 将所述第一提示词输入所述规则生成大模型，由所述规则生成大模型输出所述参考网页对应的多个第一参考规则项，各所述第一参考规则项分别为一个所述内容采集规则项或一个所述页面关系规则项；

[0152] 根据所述参考网页的类型以及所述多个第一参考规则项，生成所述数据采集规则信息。

[0153] 作为一种可能的实施方式，所述生成模块1203具体用于：

[0154] 若所述参考网页为列表页，则获取所述参考网页中至少一个列表链接所链接到的详情页的HTML文本；

[0155] 根据各详情页的HTML文本，生成各详情页对应的第二提示词；

[0156] 将所述第二提示词输入所述规则生成大模型，由所述规则生成大模型输出各所述详情页对应的多个第二参考规则项，各所述第二参考规则项分别为一个所述内容采集规则项；

[0157] 根据所述多个第一参考规则项以及各详情页对应的多个第二参考规则项，生成所述数据采集规则信息。

[0158] 作为一种可能的实施方式，所述生成模块1203具体用于：

[0159] 分别测试所述多个第一参考规则项中的各内容采集规则以及各详情页对应的各第二参考规则项的信息抽取结果；

[0160] 根据所述多个第一参考规则项中的各内容采集规则项以及各详情页对应的各第二参考规则项的信息抽取结果，从所述多个第一参考规则项中的各内容采集规则项以及各详情页对应的各第二参考规则项中选择出所述数据采集规则信息中的各所述内容采集规则项，并将所述多个第一参考规则项中的各页面关系规则项分别作为所述数据采集规则信息中的一个页面关系规则项。

[0161] 作为一种可能的实施方式，所述采集模块1024具体用于：

[0162] 基于所述数据采集规则信息对具有相同布局的多个验证网页进行数据采集，得到各验证网页的数据采集结果；

[0163] 获取具有相同布局的多个验证网页的HTML文本；

[0164] 将各验证网页的数据采集结果以及所述验证网页的HTML文本分别作为一组输入数据输入至预先训练得到的语义分析大模型中，得到分析结果；

[0165] 根据所述分析结果，对所述数据采集规则信息进行优化处理，得到目标数据采集规则信息；

[0166] 基于所述目标数据采集规则信息，对网络中的网页进行数据采集。

[0167] 作为一种可能的实施方式，所述采集模块1024具体用于：

[0168] 获取当前待提取网页的HTML文本；

[0169] 执行所述目标数据采集规则信息中的各内容采集规则项以及各页面关系规则项，以对所述当前待提取网页的HTML进行数据采集；

[0170] 将各内容采集规则项执行后所得到的第一采集结果进行保存；

[0171] 根据各页面关系规则项执行后所得到的第二采集结果，依次对与所述当前待提取网页存在跳转关系的各网页进行数据采集。

[0172] 作为一种可能的实施方式，所述页面关系规则项包括：列表链接规则项以及下一页规则项；所述采集模块1024具体用于：

[0173] 若所述列表链接规则项执行后所得到的第二采集结果不为空，则按照所述第二采集结果中的各网页链接的顺序，依次向各网页链接所指示的网页发送访问请求，以获取各网页链接所指示的网页的网页内容；

[0174] 若所述下一页规则项执行后所得到的第二采集结果不为空，则获取所述第二采集结果中下一页链接的目标位置，并通过动态下载器执行点击所述目标位置的操作，以获取所述下一页链接所指向的网页的网页链接，并向所述下一页链接所指向的网页发送访问请求，以获取所述下一页链接所指向的网页的网页内容。

[0175] 作为一种可能的实施方式，所述预处理模块1202具体用于：

[0176] 根据所述原始HTML文本，确定所述参考网页的树形结构；

[0177] 基于递归上下文策略对所述树形结构进行元素筛选，得到待保留元素集合；

[0178] 根据所述待保留元素集合，逆序遍历所述树形结构，以从所述树形结构中删除至少一个元素，得到目标树形结构；

[0179] 基于所述目标树形结构，生成所述目标HTML文本。

[0180] 本申请实施例还提供了一种电子设备130，如图13所示，为本申请实施例提供的电子设备130结构示意图，包括：处理器131、存储器132，可选的，还可以包括总线133。所述存储器132存储有所述处理器131可执行的机器可读指令（比如，图12中的装置中获取模块、预处理模块、生成模块以及采集模块对应的执行指令等），当电子设备130运行时，所述处理器131与所述存储器132之间通过总线133通信，所述机器可读指令被所述处理器131执行时执行方法实施例中的方法步骤。

[0181] 本申请实施例还提供了一种计算机可读存储介质，该计算机可读存储介质上存储有计算机程序，该计算机程序被处理器运行时执行方法实施例中的方法步骤。

[0182] 所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统和装置的具体工作过程，可以参考方法实施例中的对应过程，本申请中不再赘述。在本申请所提供的几个实施例中，应该理解到，所揭露的系统、装置和方法，可以通过其它的方式实现。以上所描述的装置实施例仅仅是示意性的，例如，所述模块的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，又例如，多个模块或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些通信接口，装置或模块的间接耦合或通信连接，可以是电性，机械或其它的形式。

[0183] 另外，在本申请各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本发明的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备（可以是个人计算机，服务器，或者网络设备等）执行本发明各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、只读存储器（ROM，Read-Only Memory）、随机存取存储器（RAM，RandomAccess Memory）、磁碟或者光盘等各种可以存储程序代码的介质。

[0184] 以上仅为本申请的具体实施方式，但本申请的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本申请揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本申请的保护范围之内。< / script>

Claims

1. A method for collecting web page data, characterized in that, include: Obtain the original Hypertext Markup Language (HTML) text of the reference webpage; The original HTML text is preprocessed to obtain the target HTML text corresponding to the reference webpage. The target HTML text includes multiple attribute tags related to data collection. A large model is generated based on the target HTML text and pre-trained rules, and data collection rule information is generated. The data collection rule information includes at least one content collection rule item and at least one page relationship rule item. Each content collection rule item is used to describe the collection rules of a content element in the webpage, and each page relationship rule item is used to describe the jump rules between webpages. Based on the aforementioned data collection rule information, data is collected from web pages on the network; The step of generating a large model based on the target HTML text and pre-trained rules, and generating data collection rule information, includes: Generate a first prompt word based on the target HTML text; The first prompt word is input into the rule to generate a large model. The rule-generated large model outputs multiple first reference rule items corresponding to the reference webpage. Each first reference rule item is either a content collection rule item or a page relationship rule item. If the reference webpage is a list page, then obtain the HTML text of the details page linked to by at least one list link in the reference webpage; Based on the HTML text of each detail page, generate a second prompt word for each detail page; The second prompt word is input into the rule to generate a large model. The rule-generated large model outputs multiple second reference rule items corresponding to each detail page. Each second reference rule item is a content collection rule item. Test the information extraction results of each content collection rule in the plurality of first reference rule items and each second reference rule item corresponding to each detail page; Based on the information extraction results of each content collection rule item in the plurality of first reference rule items and each second reference rule item corresponding to each detail page, each content collection rule item in the data collection rule information is selected from each content collection rule item in the plurality of first reference rule items and each second reference rule item corresponding to each detail page, and each page relationship rule item in the plurality of first reference rule items is respectively used as one page relationship rule item in the data collection rule information.

2. The method according to claim 1, characterized in that, The process of collecting data from web pages on the network based on the data collection rule information includes: Based on the data collection rule information, data is collected from multiple verification web pages with the same layout to obtain the data collection results for each verification web page. Retrieve the HTML text of multiple verification web pages with the same layout; The data collection results of each verification webpage and the HTML text of the verification webpage are respectively input into a pre-trained semantic analysis model to obtain the analysis results. Based on the analysis results, the data collection rule information is optimized to obtain the target data collection rule information; Based on the target data collection rule information, data is collected from web pages on the network.

3. The method according to claim 2, characterized in that, The step of collecting data from web pages on the network based on the target data collection rule information includes: Get the HTML text of the current webpage to be extracted; Execute each content collection rule item and each page relationship rule item in the target data collection rule information to collect data from the HTML of the current webpage to be extracted; Save the first collection result obtained after executing each content collection rule item; Based on the second collection result obtained after executing each page relationship rule item, data is collected sequentially from each webpage that has a jump relationship with the current webpage to be extracted.

4. The method according to claim 3, characterized in that, The page relationship rules include: list link rules and next page rules; The second collection result obtained after executing the page relationship rules is used to sequentially collect data from each webpage that has a redirection relationship with the current webpage to be extracted, including: If the second collection result obtained after executing the list link rule item is not empty, then according to the order of each webpage link in the second collection result, access requests are sent to the webpages indicated by each webpage link in turn to obtain the webpage content of each webpage link. If the second collection result obtained after executing the next page rule item is not empty, then the target position of the next page link in the second collection result is obtained, and the operation of clicking the target position is performed through the dynamic downloader to obtain the webpage link of the webpage pointed to by the next page link, and an access request is sent to the webpage pointed to by the next page link to obtain the webpage content of the webpage pointed to by the next page link.

5. The method according to claim 1, characterized in that, The preprocessing of the original HTML text to obtain the target HTML text corresponding to the reference webpage includes: Based on the original HTML text, determine the tree structure of the reference webpage; The tree structure is filtered based on a recursive context strategy to obtain a set of elements to be retained. Based on the set of elements to be retained, the tree structure is traversed in reverse order to delete at least one element from the tree structure, thereby obtaining the target tree structure; Based on the target tree structure, the target HTML text is generated.

6. An electronic device, characterized in that, include: The device includes a processor and a memory, the memory storing machine-readable instructions executable by the processor, which, when the electronic device is in operation, are executed by the processor to perform the steps of the web page data acquisition method as described in any one of claims 1 to 5.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, performs the steps of the web page data acquisition method as described in any one of claims 1 to 5.

Images