Page information extraction method and device and electronic equipment

Abstract

The invention discloses a page information extraction method and device and electronic equipment, and the method comprises the steps: obtaining a target webpage based on a resource locator, and constructing a target DOM node tree corresponding to the target webpage according to a rendering demand; inputting the target DOM node tree and a pre-edited information extraction instruction into an information extraction model to obtain to-be-filled information; and filling the to-be-filled information into the target template to generate a target document. According to the embodiment of the invention, the target webpage is obtained based on the resource locator, and the target DOM node tree corresponding to the target webpage is constructed according to the rendering requirement, so that full-amount rendering of the target webpage is effectively avoided, the rendering workload is greatly reduced, the rendering efficiency is improved, and the dynamic resource consumption is reduced. And meanwhile, information extraction is directly carried out on the target DOM node tree, so that the method can be widely applied to extraction of different page information and is more flexible.

Description

Technical Field

[0001] The present invention relates to the technical field of information extraction, and in particular to a page information extraction method, device and electronic equipment. Background Art

[0002] Extracting news information from new web pages is widely used in market research, personalized services and other fields.

[0003] Currently, news information extraction mainly relies on HTML parsing, rule engines, or full dynamic rendering. However, these methods suffer from incomplete content extraction, high resource consumption, and the need to develop specific extraction rules for different news web pages, resulting in limited flexibility.

[0004] In order to solve the above problems, it is necessary to improve the method of extracting information from news web pages. Summary of the Invention

[0005] The present invention provides a page information extraction method, device and electronic device to solve the problems of missing extracted content, inability to apply extraction rules to different pages, long information extraction time and high dynamic resource consumption in traditional page information extraction.

[0006] In a first aspect, an embodiment of the present invention provides a method for extracting page information, comprising:

[0007] Acquire a target webpage based on the resource locator, and construct a target DOM node tree corresponding to the target webpage according to rendering requirements;

[0008] Input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled;

[0009] Fill the target template with the information to be filled in to generate the target document.

[0010] In a second aspect, an embodiment of the present invention further provides a page information extraction device, comprising:

[0011] A node tree construction module is used to obtain a target web page based on a resource locator and construct a target DOM node tree corresponding to the target web page according to rendering requirements;

[0012] An information extraction module is used to input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled;

[0013] The document generation module is used to fill the information to be filled into the target template to generate a target document.

[0014] In a third aspect, an embodiment of the present invention further provides an electronic device, including:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor so that the at least one processor can perform the page information extraction method described in any embodiment of the present invention.

[0018] In a fourth aspect, an embodiment of the present invention further provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a processor to implement the page information extraction method described in any embodiment of the present invention when executed.

[0019] In a fifth aspect, an embodiment of the present invention further provides a computer program product, including a computer program, which, when executed by a processor, implements the page information extraction method as described in any one of the embodiments of the present invention.

[0020] The technical solution of the embodiment of the present invention obtains the target web page based on the resource locator, and constructs the target DOM node tree corresponding to the target web page according to the rendering requirements, so that only necessary resources are loaded, and the loading of resources such as pictures, videos or links is prohibited. It can effectively avoid the full rendering of the target web page, and only perform lightweight rendering on the content corresponding to the rendering requirements, which greatly reduces the rendering workload, improves the rendering efficiency, and reduces the dynamic resource consumption. At the same time, by directly extracting information from the target DOM node tree, there is no need to formulate different extraction rules for different web pages, but it is widely used for different page information extraction, which is more flexible. Further, the target DOM node tree and the pre-edited information extraction instructions are input into the information extraction model to obtain the information to be filled. The target DOM node tree is subjected to targeted information extraction through accurate information extraction instructions, which can improve the accuracy and efficiency of information extraction. Further, the information to be filled is filled into the target template to generate the target document, so that the format of the target document is more unified, thereby making it more convenient to apply the extracted information content to the effects of various fields.

[0021] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present invention, nor is it intended to limit the scope of the present invention. Other features of the present invention will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0023] Figure 1 This is a flowchart of a page information extraction method provided in accordance with the first embodiment of the present invention;

[0024] Figure 2 This is a flow chart of a page information extraction method provided according to the second embodiment of the present invention;

[0025] Figure 3 This is a flow chart of a page information extraction method provided according to the second embodiment of the present invention;

[0026] Figure 4 This is a schematic diagram of the structure of a page information extraction device provided according to a third embodiment of the present invention;

[0027] Figure 5 It is a structural diagram of an electronic device for implementing the page information extraction method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the solutions of the present invention, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only embodiments of a part of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention. The acquisition, transmission, storage, use, and processing of data in the technical solution of this application comply with the relevant provisions of national laws and regulations. It should be noted that in the embodiments of this application, certain software, components, or models that already exist in the industry may be mentioned. They should be regarded as exemplary. Their purpose is only to illustrate the feasibility of the implementation of the technical solution of this application, but it does not mean that the applicant has or must use the solution.

[0029] It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, such that the embodiments of the present invention described herein can be practiced in an order other than that illustrated or described herein.

[0030] Example 1

[0031] Figure 1 A flowchart of a page information extraction method is provided for embodiment 1 of the present invention. This embodiment can be applied to lightweight rendering of text information in a web page according to rendering requirements, and targeted information extraction from the web page according to information extraction instructions through an information extraction model. The method can be executed by a page information extraction device, which can be implemented in the form of hardware and / or software, and the page information extraction device can be configured in a computing device that can execute the page information extraction method.

[0032] like Figure 1 As shown, the method includes:

[0033] S110 : Acquire a target webpage based on the resource locator, and construct a target DOM node tree corresponding to the target webpage according to rendering requirements.

[0034] Among them, the resource locator refers to a standardized address system used to identify and locate resources on the Internet. The target web page can be understood as the web page corresponding to the address pointed to by the resource locator entered in the page. The target web page in this technical solution is a document page generated based on the HTML frame. The DOM node tree can be understood as a structured mapping of the target web page in memory, and page elements, text, and comments can be converted into programmable objects through the attribute data structure. The rendering requirement in this technical solution refers to the requirement to render the text information in the target web page. Accordingly, the target DOM node tree refers to a structured mapping corresponding to the text part in the target web page.

[0035] Taking web page A as an example, web page A is retrieved and displayed by inputting the resource locator (also understood as the URL) of web page A in the web page.

[0036] Taking the target web page as a news page as an example, generally speaking, in addition to text information, the target web page may also contain non-text interference such as pictures, videos, advertisements, links, browsing boxes to be selected, and controls such as "show more" or "next page". Among them, advertisements may have various expressions. For example, advertisements may appear as "related recommendations", "hot topics" or dynamic pop-ups, etc. Non-text interference can appear as comment blocks, navigation menus, recommended links, etc. If a crawler is used to obtain news information directly from the target web page, it is necessary to fully render the page to dynamically obtain all the content on the target web page. This process consumes a lot of resources and is time-consuming. It is also impossible to accurately obtain the desired text information from the complex page content.

[0037] Based on this, when the present technical solution detects that the user has entered a resource locator corresponding to the target web page in the search page, the target web page is retrieved according to the resource locator. Furthermore, in order to quickly obtain the desired text information from the target web page, the non-text information in the target web page can be filtered, and only the text content part of the target web page can be structured and mapped to generate the corresponding target DOM node tree. For example, a lightweight headless browser (such as Puppeteer lite mode) is started, only the basic HTML framework of the target web page is loaded, and the download of non-essential resources such as pictures and videos is prohibited.

[0038] The advantage of this setup is that extracting text information from the target webpage no longer requires rendering the entire page. Instead, only the corresponding DOM node tree needs to be loaded according to the rendering requirements, for example, only the DOM node tree corresponding to the text content is loaded. Based on this, this technical solution can achieve lightweight rendering of the target webpage, greatly reducing the dynamic resource consumption during the webpage rendering process and improving rendering speed.

[0039] S120 : Input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled.

[0040] The information extraction instruction refers to an instruction for extracting specified information from a target web page. The information extraction model refers to a model for extracting information corresponding to the information extraction instruction from a target DOM node tree. The information to be filled refers to the text information corresponding to the information extraction instruction.

[0041] For example, the information extraction instruction may be an instruction for extracting the title, text, or release time of the target web page. The information extraction model may be a large prediction model or a text extraction model.

[0042] Optionally, before inputting the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled, the method further includes: setting at least one information extraction instruction according to information extraction requirements.

[0043] The information extraction instructions include a title extraction instruction, a release time extraction instruction, and a text extraction instruction. The title extraction instruction is defined as extracting the title information corresponding to the first title tag in the target DOM node tree.

[0044] Taking the information extraction instruction as an example of a text extraction instruction, since the target DOM node tree contains attribute structure data and programmable objects corresponding to the text content in the target web page, based on this, the target DOM node tree and the pre-edited information extraction instruction are input into the information extraction model, and the text corresponding to the text extraction instruction can be extracted from the target DOM node tree.

[0045] It should be noted that the target webpage's main text or publication date is generally fixed and easy to identify, but the target webpage may contain multiple titles. For example, in addition to news titles, the target webpage may also include advertising titles, and even the titles in the news titles can be divided into first-level titles, second-level titles, etc. Therefore, when setting the title extraction instruction, it is necessary to further define the extraction rules. Generally speaking, when the target webpage is a news webpage, the news title to be extracted is usually the first title on the page. Therefore, the title extraction instruction can be defined as extracting the title information corresponding to the first title tag in the target DOM node tree, thereby ensuring the correct extraction of the news title.

[0046] S130: Fill the target template with the information to be filled in, and generate a target document.

[0047] The target template is a document template used to display the text information the user wishes to extract. It contains prompts corresponding to each information extraction instruction, and each prompt has a corresponding area for the information to be filled in. The target document contains the text information extracted according to the information extraction instructions. The target document is generated from the text information extracted from the target DOM node tree according to the information extraction instructions.

[0048] The technical solution of the embodiment of the present invention obtains the target web page based on the resource locator, and constructs the target DOM node tree corresponding to the target web page according to the rendering requirements, so that only necessary resources are loaded, and the loading of resources such as pictures, videos or links is prohibited. It can effectively avoid the full rendering of the target web page, and only perform lightweight rendering on the content corresponding to the rendering requirements, which greatly reduces the rendering workload, improves the rendering efficiency, and reduces the dynamic resource consumption. At the same time, by directly extracting information from the target DOM node tree, there is no need to formulate different extraction rules for different web pages, but it is widely used for different page information extraction, which is more flexible. Further, the target DOM node tree and the pre-edited information extraction instructions are input into the information extraction model to obtain the information to be filled. The target DOM node tree is subjected to targeted information extraction through accurate information extraction instructions, which can improve the accuracy and efficiency of information extraction. Further, the information to be filled is filled into the target template to generate the target document, so that the format of the target document is more unified, thereby making it more convenient to apply the extracted information content to the effects of various fields.

[0049] Example 2

[0050] Figure 2 This is a flowchart of a page information extraction method provided in the second embodiment of the present invention. Optionally, the target DOM node tree corresponding to the target web page constructed according to rendering requirements is refined.

[0051] like Figure 2 As shown, the method includes:

[0052] S210: Acquire a target web page based on the resource locator.

[0053] In a specific example, taking the target web page as a news web page, such as Figure 3 As shown, a link (ie, a resource locator) corresponding to a target web page is input to obtain the target web page.

[0054] S220: Determine the page display type of the target webpage according to the preset control information in the target webpage.

[0055] The preset control information refers to a control or trigger area used to display all browsing information in the target web page.

[0056] Generally speaking, when there is a lot of content in the target web page, it is impossible to present all the content to the user in the current display area. At this time, the target web page will generally be set with preset control information, such as "Load More", "Read Full Text" or "Next Page". Therefore, the page display types can be roughly divided into two categories, single-page type and multi-page type. For example, when the preset control information is "Load More" or "Read Full Text", all the content is essentially on the same display page, and when the preset control information is "Next Page", the content of the target web page essentially needs to be displayed on multiple pages.

[0057] The advantage of this setting is that by adopting different information extraction methods for different page types, different page types can be effectively dealt with. That is, regardless of whether the text information to be extracted is the content of a single page or the content of multiple pages, the method of this technical solution can be used to accurately extract the text content, and the application is more flexible and extensive.

[0058] S230: Determine a basic DOM node tree corresponding to the target web page based on a trigger operation corresponding to the page display type.

[0059] The basic DOM node tree refers to a structured mapping containing all elements in the target web page.

[0060] For example, the basic DOM node tree includes a structured mapping corresponding to the text information that the user wants to extract, as well as result-based mappings corresponding to elements such as pictures, videos, advertisements, links, and browsing boxes to be selected. In other words, the basic DOM node tree is a structured mapping corresponding to the original state of the target web page.

[0061] Still taking the target web page as a news web page as an example, based on the key interactive elements in the news web page, such as "Load More" (i.e., preset control information), it is determined that the page display type of the news web page is a single page type. At this time, it is necessary to trigger "Load More" in the page to display all the information in the news web page and obtain the basic DOM node tree corresponding to the news web page.

[0062] In another example, if the triggering of the "next page" (ie, preset control information) in the news webpage is detected, the resource locator corresponding to the news webpage will usually be updated. At this time, the page display type of the news webpage is determined to be a multi-page type.

[0063] Based on this, there are two ways to construct the basic DOM node tree for the news page. The first is to trigger the "Next Page" function on each news page and build a DOM node tree corresponding to each page. After the DOM node trees for all pages are built, they are spliced ​​together to obtain the basic DOM node tree corresponding to the news page. The second method is to merge the page information of all pages into a single page when the preset control information is "Next Page" and build a DOM node tree corresponding to the entire news page as the basic DOM node tree.

[0064] S240: Simplify the basic DOM node tree based on rendering requirements to obtain a target DOM node tree.

[0065] Based on the above example, continue to see Figure 3 , mainly for extracting text information from web pages. For example, if the target web page is a news page, the text information in the news page is extracted. Therefore, when simplifying the basic DOM node tree according to rendering requirements, only the structured mapping corresponding to the text information is loaded, and redundant tags in the basic DOM node tree are removed, achieving dynamic web page loading and lightweight rendering, and obtaining the target DOM node tree.

[0066] For example, start a lightweight headless browser (such as Puppeteer Lite mode), load only the basic HTML framework (that is, the basic DOM node tree), prohibit the download of non-essential resources such as pictures and videos, and clean the HTML. <script>、<style>、<svg>等非文本标签；剥离图片Base64编码、CSS样式属性等噪声数据，仅保留文本内容及语义化标签(如<h1>-<h6>、、)，得到目标DOM节点树。

[0067] 可选的，基于渲染需求简化基础DOM节点树，得到目标DOM节点树，包括：根据渲染需求，确定目标网页中的必要加载资源；根据必要加载资源，从基础DOM节点树中提取与必要加载资源对应的目标节点，得到待处理DOM节点树；对待处理DOM节点树进行清洗处理和简化处理，得到目标DOM节点树。

[0068] 其中，目标节点包含必要加载资源的渲染信息。所谓的必要加载资源是指用户期望的从目标网页中获取的信息。目标节点是指与必要加载资源对应的结构化标签。待处理DOM节点树是指基础DOM节点树中与必要加载资源对应的结构化映射。

[0069] 例如，目标网页是新闻网页，则必要加载资源是新闻网页中的文本信息，目标节点可以是新闻网页对应的基础DOM节点树中与文本信息对应的文本标签和属性数据节点等。待处理DOM节点树为从基础DOM节点树中提取的与文本信息对应的结构化映射。

[0070] 在上述示例的基础上，对待处理DOM节点树进行清洗处理和简化处理，得到目标DOM节点树，包括：基于预设剔除规则，清洗待处理DOM节点树中的冗余信息；对待处理DOM节点树进行层级压缩，并保留待处理DOM节点树中的关键属性，得到目标DOM节点树。

[0071] 其中，预设剔除规则包括标签剔除规则和预设数据格式对应的剔除规则；，如，预设剔除规则包括去除冗余标签规则或去除非文本内容规则等，具体的可以根据实际情况设定。

[0072] 在上述示例的基础上，继续参见图3，基于预设剔除规则对待处理DOM节点树执行清洗HTML中的<script>、<style>、<svg>等非文本标签，同时剥离图片Base64编码、CSS样式属性等噪声数据，仅保留文本内容及语义化标签(如<h1>-<h6>、、)的操作。进一步的，对于待处理DOM节点树中的嵌套层级深的节点进行层级压缩，以简化待处理DOM节点树的层级结构。如，将div>div>div>p简化为div>p，即，将三级嵌套层级简化为单层层级。同时，保留待处理DOM节点树中的关键属性作为语义标记，如id＝"article-title"、class＝"publish-time")。

[0073] 这样设置的好处在于，以目标网页为新闻网页为例，当待处理DOM节点树中的嵌套层级太多时，若想从节点树中提取期望的文本信息需要逐层的进行解析，在此过程中会存在较多的冗余操作，较为耗时，而将嵌套在多层级中的文本信息直接简化为单层级结构后，对要提取的文本信息可以直接进行解析，从而可以提高文本信息的提取效率。

[0074] S250、将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息。

[0075] 在上述示例的基础上，需要从新闻网页中提取的内容包括：

[0076] (1)标题：从<h1>或<h2>标签中提取新闻主标题，排除广告标语；

[0077] (2)发布时间：识别时间格式文本(如YYYY-MM-DD HH:mm)，优先选择含"publish""time"类名的节点；

[0078] (3)正文：聚合所有标签内容，过滤class含"ad""banner"的节点；

[0079] (4)广告列表：标记所有含推广语义的文本块。

[0080] 基于此，设置与提取内容对应的信息提取指令，如下：

[0081] 标题提取指令：定位最高层级的<h1>或<h2>，排除含"广告""推荐"的节点；

[0082] 发布时间提取指令：匹配标准化时间格式，忽略"更新时间""编辑时间"等干扰项；

[0083] 正文提取指令：合并所有标签文本，自动补全动态加载的分页内容；

[0084] 广告过滤指令：标记并移除class含ad、sponsored或文本含"推广""特惠"的区块。

[0085] 在此基础上，将目标DOM节点树和信息提取指令输入LLM(即，信息提取模型)以得到待填充信息。

[0086] 这样设置好处在于，通过明确的信息提取指令有针对性的对新闻网页对应的待处理DOM节点树中的内容进行提取，解决了传统页面信息提取的方法中的盲目性问题，提高信息提取的准确率和提取效率。

[0087] S260、将待填充信息填充至目标模板，生成目标文档。

[0088] 可选的，将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息，包括：将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待校验信息；对待校验信息进行关键字段检测，若待校验信息中包含所有信息提取指令对应的文本内容，得到待填充信息。

[0089] 其中，待校验信息根据信息提取指令从目标DOM节点树中提取的文本信息。

[0090] 需要说明的是，在从目标DOM节点树中提取与信息提取指令对应的文本信息时，可能存在关键信息缺失的情况。例如，继续参见图3，根据标题提取指令、发布时间提取指令和正文提取指令从目标网页中提取文本信息，并将提取的文本信息作为待校验信息。为了确定待校验信息中是否包含所有的信息提取指令对应的文本信息，需要进一步的对待校验信息进行校验以保证文本信息提取的完整性。若待校验信息中包含与信息提取指令对应的标题、发布时间和正文，则表明待校验信息中不缺少关键文本内容，可以将待校验信息作为待填充信息，以将待填充信息填写至目标模板中的相应位置得到目标文档。

[0091] 反之，若待校验信息中缺少关键文本内容，则根据关键文本内容的文本标签从目标DOM节点树中提取关键文本内容，并将关键文本内容添加至待填充信息。

[0092] 其中，关键文本内容是指待校验信息中应包含，但未包含在内的与信息提取指令对应的文本内容。以上述示例为例，在信息提取模型中输入的信息提取指令有标题提取指令、发布时间提取指令和正文提取指令，待校验信息中只包含标题和正文，未包含发布时间，则此时将发布时间称为关键文本内容。

[0093] 具体的，对待校验信息进行关键字段检测，若在目标网页对应的目标DOM节点树中缺少发布时间，则此时存在关键信息缺失的情况，此时可以根据"发布时间”这个文本标签从目标DOM节点树中提取相应的文本内容(即，基于DOM规则补全)，并将关键文本内容添加至待填充信息。例如，提取<meta property＝"article:published_time">以从目标DOM节点树中提取出发布时间。

[0094] 在此基础上，继续参见图3，对待填充信息进行格式转换，并将转换后的待填充信息填充至目标模板的相应位置进行标准化格式输出，生成目标文档。

[0095] 示例性地，对待填充信息转换为结构化数据，例如：

[0096] {

[0097] "title":"某国通过新气候法案",

[0098] "publish_time":"2024-07-25 10:30",

[0099] "content":"正文第一段..."

[0100] }

[0101] 进一步的，对得到的结构化数据进行以下操作，以使待填充信息满足格式要求：

[0102] 时间修正：将非标准时间(如"7 / 25 / 2024")转为YYYY-MM-DD格式；

[0103] 标题净化：去除冗余前缀(如"快讯：")；

[0104] 正文去重：删除因动态加载导致的重复段落。

[0105] 在此基础上，将转换后的待填充信息填充至目标模板中的相应位置，得到目标文档。

[0106] 本发明实施例的技术方案，通过基于资源定位符获取目标网页，并根据渲染需求构建与目标网页对应的目标DOM节点树，从而仅加载必要加载资源，禁止图片、视频或链接等的资源加载，可以有效避免对目标网页的全量渲染，仅对渲染需求对应的内容进行轻量化渲染，大大降低了渲染工作量，达到了提高渲染效率，降低动态资源消耗。同时，通过直接对目标DOM节点树进行信息提取，不需要针对不同的网页制定不同的提取规则，而是广泛应用于不同的页面信息提取，更加灵活。进一步的，将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息，通过准确的信息提取指令对目标DOM节点树进行有针对性的信息提取，可以提高信息提取的准确性和提取效率。进一步的，将待填充信息填充至目标模板，生成目标文档，使得目标文档的格式更加统一，从而更加方便的将提取的信息内容应用于各领域的效果。

[0107] 实施例三

[0108] 图4为本发明实施例三提供的一种页面信息提取装置的结构示意图。如图4所示，该装置包括：节点树构建模块310、信息提取模块320和文档生成模块330。

[0109] 其中，节点树构建模块310，用于基于资源定位符获取目标网页，并根据渲染需求构建与目标网页对应的目标DOM节点树；

[0110] 信息提取模块320，用于将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息；

[0111] 文档生成模块330，用于将待填充信息填充至目标模板，生成目标文档。

[0112] 本发明实施例的技术方案，通过基于资源定位符获取目标网页，并根据渲染需求构建与目标网页对应的目标DOM节点树，从而仅加载必要加载资源，禁止图片、视频或链接等的资源加载，可以有效避免对目标网页的全量渲染，仅对渲染需求对应的内容进行轻量化渲染，大大降低了渲染工作量，达到了提高渲染效率，降低动态资源消耗。同时，通过直接对目标DOM节点树进行信息提取，不需要针对不同的网页制定不同的提取规则，而是广泛应用于不同的页面信息提取，更加灵活。进一步的，将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息，通过准确的信息提取指令对目标DOM节点树进行有针对性的信息提取，可以提高信息提取的准确性和提取效率。进一步的，将待填充信息填充至目标模板，生成目标文档，使得目标文档的格式更加统一，从而更加方便的将提取的信息内容应用于各领域的效果。

[0113] 可选的，节点树构建模块包括：类型确定子模块，用于根据目标网页中的预设控件信息，确定目标网页的页面展示类型；其中，页面展示类型包括多页面展示类型或单页面展示类型；

[0114] 基础节点树确定子模块，用于基于页面展示类型对应的触发操作，确定目标网页对应的基础DOM节点树；

[0115] 目标节点树确定子模块，用于基于渲染需求简化基础DOM节点树，得到目标DOM节点树。

[0116] 可选的，目标节点树确定子模块包括：资源确定单元，用于根据渲染需求，确定目标网页中的必要加载资源；

[0117] 待处理节点树确定单元，用于根据必要加载资源，从基础DOM节点树中提取与必要加载资源对应的目标节点，得到待处理DOM节点树；其中，目标节点包含必要加载资源的渲染信息；

[0118] 目标节点树确定单元，用于对待处理DOM节点树进行清洗处理和简化处理，得到目标DOM节点树。

[0119] 可选的，目标节点树确定单元包括：剔除子单元，用于基于预设剔除规则，清洗待处理DOM节点树中的冗余信息；其中，预设剔除规则标签剔除规则和预设数据格式对应的剔除规则；

[0120] 目标节点树确定子单元，用于对待处理DOM节点树进行层级压缩，并保留待处理DOM节点树中的关键属性，得到目标DOM节点树。

[0121] 可选的，页面信息提取装置，还包括，指令设置模块，用于在将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待填充信息之前，根据信息提取需求，设置至少一个信息提取指令；其中，信息提取指令包含标题提取指令、发布时间提取指令和正文提取指令，标题提取指令定义为提取目标DOM节点树中的首个标题标签对应的标题信息。

[0122] 可选的，信息提取模块，包括：待校验信息确定子模块，用于将目标DOM节点树和预先编辑的信息提取指令输入信息提取模型，得到待校验信息；

[0123] 信息提取子模块，用于对待校验信息进行关键字段检测，若待校验信息中包含所有信息提取指令对应的文本内容，得到待填充信息。

[0124] 页面信息提取装置，还包括：添加模块，用于若待校验信息中缺少关键文本内容，则根据关键文本内容的文本标签从目标DOM节点树中提取关键文本内容，并将关键文本内容添加至待填充信息。

[0125] 文档生成模块，用于对待填充信息进行格式转换，并将转换后的待填充信息填充至目标模板的相应位置，生成目标文档。

[0126] 本发明实施例所提供的页面信息提取装置可执行本发明任意实施例所提供的页面信息提取方法，具备执行方法相应的功能模块和有益效果。

[0127] 实施例四

[0128] 图5示出了本发明的实施例的电子设备10的结构示意图。电子设备旨在表示各种形式的数字计算机，诸如，膝上型计算机、台式计算机、工作台、个人数字助理、服务器、刀片式服务器、大型计算机、和其它适合的计算机。电子设备还可以表示各种形式的移动装置，诸如，个人数字处理、蜂窝电话、智能电话、可穿戴设备(如头盔、眼镜、手表等)和其它类似的计算装置。本文所示的部件、它们的连接和关系、以及它们的功能仅仅作为示例，并且不意在限制本文中描述的和 / 或者要求的本发明的实现。

[0129] 如图5所示，电子设备10包括至少一个处理器11，以及与至少一个处理器11通信连接的存储器，如只读存储器(ROM)12、随机访问存储器(RAM)13等，其中，存储器存储有可被至少一个处理器执行的计算机程序，处理器11可以根据存储在只读存储器(ROM)12中的计算机程序或者从存储单元18加载到随机访问存储器(RAM)13中的计算机程序，来执行各种适当的动作和处理。在RAM 13中，还可存储电子设备10操作所需的各种程序和数据。处理器11、ROM 12以及RAM 13通过总线14彼此相连。输入 / 输出(I / O)接口15也连接至总线14。

[0130] 电子设备10中的多个部件连接至I / O接口15，包括：输入单元16，例如键盘、鼠标等；输出单元17，例如各种类型的显示器、扬声器等；存储单元18，例如磁盘、光盘等；以及通信单元19，例如网卡、调制解调器、无线通信收发机等。通信单元19允许电子设备10通过诸如因特网的计算机网络和 / 或各种电信网络与其他设备交换信息 / 数据。

[0131] 处理器11可以是各种具有处理和计算能力的通用和 / 或专用处理组件。处理器11的一些示例包括但不限于中央处理单元(CPU)、图形处理单元(GPU)、各种专用的人工智能(AI)计算芯片、各种运行机器学习模型算法的处理器、数字信号处理器(DSP)、以及任何适当的处理器、控制器、微控制器等。处理器11执行上文所描述的各个方法和处理，例如页面信息提取方法。

[0132] 在一些实施例中，页面信息提取方法可被实现为计算机程序，其被有形地包含于计算机可读存储介质，例如存储单元18。在一些实施例中，计算机程序的部分或者全部可以经由ROM 12和 / 或通信单元19而被载入和 / 或安装到电子设备10上。当计算机程序加载到RAM 13并由处理器11执行时，可以执行上文描述的页面信息提取方法的一个或多个步骤。备选地，在其他实施例中，处理器11可以通过其他任何适当的方式(例如，借助于固件)而被配置为执行页面信息提取方法。

[0133] 本文中以上描述的系统和技术的各种实施方式可以在数字电子电路系统、集成电路系统、现场可编程门阵列(FPGA)、专用集成电路(ASIC)、专用标准产品(ASSP)、系统级芯片(SOC)、复杂可编程逻辑设备(CPLD)、计算机硬件、固件、软件、和 / 或它们的组合中实现。这些各种实施方式可以包括：实施在一个或者多个计算机程序中，该一个或者多个计算机程序可在包括至少一个可编程处理器的可编程系统上执行和 / 或解释，该可编程处理器可以是专用或者通用可编程处理器，可以从存储系统、至少一个输入装置、和至少一个输出装置接收数据和指令，并且将数据和指令传输至该存储系统、该至少一个输入装置、和该至少一个输出装置。

[0134] 用于实施本发明的页面信息提取方法的计算机程序可以采用一个或多个编程语言的任何组合来编写。这些计算机程序可以提供给通用计算机、专用计算机或其他可编程数据处理装置的处理器，使得计算机程序当由处理器执行时使流程图和 / 或框图中所规定的功能 / 操作被实施。计算机程序可以完全在机器上执行、部分地在机器上执行，作为独立软件包部分地在机器上执行且部分地在远程机器上执行或完全在远程机器或服务器上执行。

[0135] 在本发明的上下文中，计算机可读存储介质可以是有形的介质，其可以包含或存储以供指令执行系统、装置或设备使用或与指令执行系统、装置或设备结合地使用的计算机程序。计算机可读存储介质可以包括但不限于电子的、磁性的、光学的、电磁的、红外的、或半导体系统、装置或设备，或者上述内容的任何合适组合。备选地，计算机可读存储介质可以是机器可读信号介质。机器可读存储介质的更具体示例会包括基于一个或多个线的电气连接、便携式计算机盘、硬盘、随机存取存储器(RAM)、只读存储器(ROM)、可擦除可编程只读存储器(EPROM或快闪存储器)、光纤、便捷式紧凑盘只读存储器(CD-ROM)、光学储存设备、磁储存设备、或上述内容的任何合适组合。

[0136] 为了提供与用户的交互，可以在电子设备上实施此处描述的系统和技术，该电子设备具有：用于向用户显示信息的显示装置(例如，CRT(阴极射线管)或者LCD(液晶显示器)监视器)；以及键盘和指向装置(例如，鼠标或者轨迹球)，用户可以通过该键盘和该指向装置来将输入提供给电子设备。其它种类的装置还可以用于提供与用户的交互；例如，提供给用户的反馈可以是任何形式的传感反馈(例如，视觉反馈、听觉反馈、或者触觉反馈)；并且可以用任何形式(包括声输入、语音输入或者、触觉输入)来接收来自用户的输入。

[0137] 可以将此处描述的系统和技术实施在包括后台部件的计算系统(例如，作为数据服务器)、或者包括中间件部件的计算系统(例如，应用服务器)、或者包括前端部件的计算系统(例如，具有图形用户界面或者网络浏览器的用户计算机，用户可以通过该图形用户界面或者该网络浏览器来与此处描述的系统和技术的实施方式交互)、或者包括这种后台部件、中间件部件、或者前端部件的任何组合的计算系统中。可以通过任何形式或者介质的数字数据通信(例如，通信网络)来将系统的部件相互连接。通信网络的示例包括：局域网(LAN)、广域网(WAN)、区块链网络和互联网。

[0138] 计算系统可以包括客户端和服务器。客户端和服务器一般远离彼此并且通常通过通信网络进行交互。通过在相应的计算机上运行并且彼此具有客户端-服务器关系的计算机程序来产生客户端和服务器的关系。服务器可以是云服务器，又称为云计算服务器或云主机，是云计算服务体系中的一项主机产品，以解决了传统物理主机与VPS服务中，存在的管理难度大，业务扩展性弱的缺陷。

[0139] 实施例五

[0140] 本发明实施例还提供了一种计算机程序产品，包括计算机程序，该计算机程序在被处理器执行时实现如本申请任一实施例所提供的页面信息提取方法。

[0141] 计算机程序产品在实现的过程中，可以以一种或多种程序设计语言或其组合来编写用于执行本发明操作的计算机程序代码，程序设计语言包括面向对象的程序设计语言，诸如Java、Smalltalk、C++，还包括常规的过程式程序设计语言，诸如"C”语言或类似的程序设计语言。程序代码可以完全地在用户计算机上执行、部分地在用户计算机上执行、作为一个独立的软件包执行、部分在用户计算机上部分在远程计算机上执行、或者完全在远程计算机或服务器上执行。在涉及远程计算机的情形中，远程计算机可以通过任意种类的网络——包括局域网(LAN)或广域网(WAN)—连接到用户计算机，或者，可以连接到外部计算机(例如利用因特网服务提供商来通过因特网连接)。

[0142] 应该理解，可以使用上面所示的各种形式的流程，重新排序、增加或删除步骤。例如，本发明中记载的各步骤可以并行地执行也可以顺序地执行也可以不同的次序执行，只要能够实现本发明的技术方案所期望的结果，本文在此不进行限制。

[0143] 上述具体实施方式，并不构成对本发明保护范围的限制。本领域技术人员应该明白的是，根据设计要求和其他因素，可以进行各种修改、组合、子组合和替代。任何在本发明的精神和原则之内所作的修改、等同替换和改进等，均应包含在本发明保护范围之内。< / script>

Claims

1. A page information extraction method, characterized in that: include: Acquire a target webpage based on the resource locator, and construct a target DOM node tree corresponding to the target webpage according to rendering requirements; Input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled; Fill the target template with the information to be filled in to generate the target document.

2. The method according to claim 1, characterized in that The step of constructing a target DOM node tree corresponding to the target webpage according to the rendering requirements includes: Determining a page display type of the target webpage according to preset control information in the target webpage; wherein the page display type includes a multi-page display type or a single-page display type; Determine the basic DOM node tree corresponding to the target web page based on the trigger operation corresponding to the page display type; The basic DOM node tree is simplified based on rendering requirements to obtain a target DOM node tree.

3. The method according to claim 2, characterized in that The step of simplifying the basic DOM node tree based on the rendering requirements to obtain the target DOM node tree includes: Determining necessary loading resources in the target web page according to rendering requirements; According to the necessary loaded resources, extract the target node corresponding to the necessary loaded resources from the basic DOM node tree to obtain a DOM node tree to be processed; wherein the target node contains rendering information of the necessary loaded resources; The DOM node tree to be processed is cleaned and simplified to obtain a target DOM node tree.

4. The method according to claim 3, characterized in that The cleaning and simplifying of the DOM node tree to be processed to obtain a target DOM node tree includes: Based on preset elimination rules, the redundant information in the DOM node tree to be processed is cleaned; wherein the preset elimination rules include tag elimination rules and elimination rules corresponding to preset data formats; The DOM node tree to be processed is hierarchically compressed, and key attributes in the DOM node tree to be processed are retained to obtain a target DOM node tree.

5. The method according to claim 1, wherein Before inputting the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled, the method further includes: At least one information extraction instruction is set according to information extraction requirements; wherein, the information extraction instruction includes a title extraction instruction, a release time extraction instruction and a text extraction instruction, and the title extraction instruction is defined as extracting the title information corresponding to the first title tag in the target DOM node tree.

6. The method according to claim 1, characterized in that The step of inputting the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled includes: Input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be verified; A key field detection is performed on the information to be verified. If the information to be verified contains text content corresponding to all information extraction instructions, the information to be filled is obtained.

7. The method according to claim 6, characterized in that Also includes: If the key text content is missing from the information to be verified, the key text content is extracted from the target DOM node tree according to the text tag of the key text content, and the key text content is added to the information to be filled.

8. The method according to claim 1, characterized in that Filling the target template with the information to be filled in to generate the target document includes: The format of the information to be filled is converted, and the converted information to be filled is filled into the corresponding position of the target template to generate a target document.

9. A page information extraction device, characterized in that: include: A node tree construction module is used to obtain a target web page based on a resource locator and construct a target DOM node tree corresponding to the target web page according to rendering requirements; An information extraction module is used to input the target DOM node tree and the pre-edited information extraction instruction into the information extraction model to obtain the information to be filled; The document generation module is used to fill the information to be filled into the target template to generate a target document.

10. An electronic device, characterized in that: The electronic device comprises: at least one processor; and a memory communicatively connected to the at least one processor; wherein, The memory stores a computer program executable by the at least one processor. The computer program is executed by the at least one processor to enable the at least one processor to perform the page information extraction method according to any one of claims 1 to 8.

Images