A method, device, system and storage medium for structuring webpage content

Abstract

The present application discloses a webpage content structured processing method, device, system and storage medium, which relates to the technical field of webpage content processing. The method includes obtaining a template URL, requesting the URL address to obtain a first HTML tag; parsing and converting it into a Document object, extracting an Elements object set; removing unparsed tags, obtaining a filtered Elements object set, looping, setting a CurrentId value; defining a Tag object and assembling a List <tag>The tree structure object is constructed by reducing the node depth and integrating the node optimization method, and the tree structure is further simplified, so as to achieve the goal of extracting structured data from the web page and saving it in the database. The embodiment of the present application simplifies the complexity of data extraction, improves the efficiency of data extraction, reduces the storage space occupied, and reduces the user's learning cost and usage threshold.< / tag>

Description

Technical Field

[0001] The present application belongs to the technical field of web page content processing, and in particular, relates to a method, device, system and storage medium for structuring web page content processing. Background Art

[0002] At this stage, with the rapid development of Internet technology, web information has basically become the main way for humans to obtain news and information.

[0003] Web page information mainly relies on HTML, CSS, and SCRIPT technologies to transmit and display content data. HTML tags are diverse and flexible to use, resulting in complex and changeable web page structures. The content structures of pages on different websites vary greatly. It is difficult to obtain or extract web page content in a common way and perform structured analysis of the content.

[0004] Traditional page parsing basically uses XPath or regular expressions. XPath is a language used to select nodes in XML and HTML documents. It can navigate and locate specific element nodes in XML / HTML documents through path expressions. Regular expressions are suitable for simple text matching and extraction, processing uncomplicated HTML structures or extracting simple format data from text. Both methods require users to have certain professional knowledge and be familiar with related syntax and programming skills. For users who just use it, the professional level is high, and the content extraction and structuring methods are not intuitive. Moreover, these two methods are highly targeted and cannot dynamically match and process page data of different structures. Summary of the invention

[0005] The purpose of this application is to provide a method, device, system and storage medium for structuring the content of a web page, which further simplifies the tree structure by reducing the node depth and optimizing the node integration method, thereby achieving the goal of extracting structured data from a web page and saving it in a database.

[0006] In order to achieve the above objectives, the solution of this application is:

[0007] In a first aspect, an embodiment of the present application provides a method for structuring webpage content, comprising:

[0008] Get the template URL, request the URL address of the template URL, and get the first HTML tag; parse the first HTML tag, convert the first HTML tag into a Document object, and extract the Elements object collection in the Document object;

[0009] Remove the non-parsed tags in the Elements object collection to obtain a filtered Elements object collection. The non-parsed tags include link, script, and style tags.

[0010] Loop through the filtered Elements object collection and set the CurrentId value for each object in the filtered Elements object collection. The CurrentId value is the sequence number of each object corresponding to the loop.

[0011] Define Tag objects and populate List <tag>Collection, build a tree structure object, the Tag object includes id, parentId, name, text, src, href attributes, where id represents the tag id, and the id value is equal to the CurrentId value; parentId represents the parent tag id, and the parentId value is equal to the CurrentId value of the parent tag. If the parent tag does not exist, the parentId value is 0; name represents the tag name; src represents the src attribute value of the current tag. If the attribute does not exist, the src attribute value is Null; href represents the href attribute value of the current tag. If the attribute does not exist, the href attribute value is Null; according to List <tag>The collection, id and parentId are constructed as List <Tree <string>>Tree structure object;

[0012] Parsing the tree structure object, generating tree structure object data, the tree structure object data including node data, screening and naming the node data, forming template data and attribute names of the template data, and saving the template data into a template database;

[0013] Parse the content of the same site and match the template data. If the template data cannot be matched, the parsing fails and is recorded. Match the second HTML tag according to the attribute name of the template data. If the second HTML tag includes an href attribute or a src attribute, determine whether the value of the href attribute or the src attribute is a file or an external link. If it is a file, download the file according to the link in the second HTML tag. If it is an external link, save the link in the second HTML tag.

[0014] The CurrentId value of each object in the filtered Elements object collection, the tree structure object data, and the files or links obtained when parsing the content of similar sites are combined with the attribute name of the template data to assemble into structured object data, and the structured object data is saved to the collection database.

[0015] According to the above method of the embodiment of the present application, the following additional technical features may also be provided:

[0016] Furthermore, the Tag object includes an id attribute, a parentId attribute, a name attribute, a text attribute, a src attribute, and an href attribute.

[0017] Furthermore, before parsing the tree structure object, the method also includes reducing the node depth in the tree structure object according to a preset rule.

[0018] Furthermore, the tree structure object includes tree nodes, and the node depth in the tree structure object is reduced by a preset rule, including:

[0019] Determine whether the tree node has child nodes. If there are no child nodes, it is the end node. When the text, src, and href of the end node are empty at the same time, delete the node object and trace back upward along the hierarchical structure for iterative processing.

[0020] Furthermore, we trace back from the end node. When the number of nodes is 1 and the content contained is the same as the plain text content directly obtained from the parent node, we remove the node, use its parent node, and trace back upward along the hierarchical structure for iterative processing.

[0021] When the number of the node and its sibling nodes and the content of each node satisfy the following formula:

[0022]

[0023] in, Represents the value of the parent node. Indicates the number of child nodes. Indicates the preset threshold value,

[0024] Indicates that the value of the parent node divided by the number of child nodes is less than the preset threshold. Represents the value of the largest child node, Indicates the value of the smallest child node;

[0025] Indicates that all child nodes do not include numbers. For each node ,pass Indicates that the node contains numbers. If ,but Including numbers, if ,but Do not include numbers;

[0026] Integrate the content of this node and its sibling nodes into its parent node, use the original content of its parent node, and trace back upward along the hierarchical structure for iterative processing.

[0027] In a second aspect, an embodiment of the present application provides a webpage content structuring processing device, including:

[0028] The element acquisition module is configured to acquire the template URL, request the URL address of the template URL, and obtain the first HTML tag; parse the first HTML tag, convert the first HTML tag into a Document object, and extract the Elements object collection in the Document object;

[0029] The filtering loop module is configured to remove non-parsed tags in the Elements object set to obtain a filtered Elements object set, where the non-parsed tags include link, script and style tags; loop the filtered Elements object set, and set a CurrentId value for each object in the filtered Elements object set, where the CurrentId value is a sequence number corresponding to the loop of each object;

[0030] The tree structure module is configured to define Tag objects and assemble List <tag>Collection, build a tree structure object, the Tag object includes id, parentId, name, text, src, href attributes, where id represents the tag id, and the id value is equal to the CurrentId value; parentId represents the parent tag id, and the parentId value is equal to the CurrentId value of the parent tag. If the parent tag does not exist, the parentId value is 0; name represents the tag name; src represents the src attribute value of the current tag. If the attribute does not exist, the src attribute value is Null; href represents the href attribute value of the current tag. If the attribute does not exist, the href attribute value is Null; according to List <tag>The collection, id and parentId are constructed as List <Tree <string>>Tree structure object;

[0031] Parsing the tree structure object, generating tree structure object data, the tree structure object data including node data, screening and naming the node data, forming template data and attribute names of the template data, and saving the template data into a template database;

[0032] The parsing processing module is configured to parse the content of the same site and match the template data. If the template data cannot be matched, the parsing fails and is recorded; match the second HTML tag according to the attribute name of the template data, if the second HTML tag includes an href attribute or a src attribute, determine whether the value of the href attribute or the src attribute is a file or an external link, if it is a file, download the file according to the link in the second HTML tag; if it is an external link, save the link in the second HTML tag;

[0033] The structuring module is configured to assemble the CurrentId value of each object in the filtered Elements object collection, the tree structure object data, and the files or links obtained when parsing the content of the same site into structured object data in combination with the attribute name of the template data, and save the structured object data to the collection database.

[0034] In a third aspect, an embodiment of the present application provides a web page content structuring processing system, the system includes a processor and a memory, the memory stores a computer program, and the computer program is loaded and executed by the processor to implement the web page content structuring processing method provided in the first aspect of the embodiment of the present application.

[0035] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the web page content structuring processing method as described in the first aspect of the embodiment of the present application.

[0036] Compared with the prior art, the webpage content structured processing method provided by the embodiment of the present application has the following beneficial technical effects:

[0037] The embodiment of the present application converts unstructured web page content into structured data by obtaining the template URL and parsing HTML tags. In this process, by removing unparsed tags (such as link, script, style, etc.) and setting the CurrentId value, the complexity of data extraction is effectively simplified and the efficiency of data extraction is improved.

[0038] In this embodiment of the application, a Tag object is defined and a List is assembled. <tag>The method realizes the hierarchical representation of web page content by constructing a tree structure object. This hierarchical data structure not only facilitates the understanding and analysis of web page content, but also helps to optimize data storage and query efficiency. At the same time, the node depth in the tree structure object is reduced by preset rules, which further simplifies the data structure and reduces the storage space occupied.

[0039] When parsing the content of similar sites, the embodiment of the present application ensures the accuracy of data parsing by matching template data. If the template data cannot be matched, it is recorded and corresponding processing measures are taken. In addition, when parsing HTML tags, it is determined whether it is a file or an external link based on the value of the href attribute or the src attribute, and the download or save link processing is performed respectively, which further enhances the accuracy and integrity of data parsing.

[0040] The embodiment of the present application allows users to directly select the required data from the generated JSON tree structure and name the fields without the need for professional HTML or programming knowledge. This intuitive operation method greatly reduces the user's learning cost and usage threshold; the template nodes can be freely combined and modified according to actual needs. This high flexibility enables the method to adapt to various complex web page structures and parsing requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 A schematic diagram showing a flow chart of a webpage content structuring processing method according to an embodiment of the present application;

[0042] Figure 2 A structural block diagram of a webpage content structuring processing device according to an embodiment of the present application is shown;

[0043] Figure 3 A structural block diagram of a computer device according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. It is to be understood that the specific embodiments described herein are only used to explain the present application, rather than to limit the present application. It should also be noted that, for ease of description, only some structures related to the present application are shown in the accompanying drawings, rather than all structures. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0045] The terms "including" and "having" and any variations thereof in this application are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products or devices.

[0046] Reference to "embodiments" in this application means that a particular feature, structure, or characteristic described in conjunction with the embodiments may be included in at least one embodiment of the present application. The appearance of the phrase in various locations in the specification does not necessarily refer to the same embodiment, nor is it an independent or alternative embodiment that is mutually exclusive with other embodiments. It is explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments.

[0047] like Figure 1 As shown, the embodiment of the present application provides a method for structuring webpage content, comprising the following steps:

[0048] Step 101, obtain a template URL, request the URL address of the template URL, obtain a first HTML tag; parse the first HTML tag, convert the first HTML tag into a Document object, and extract an Elements object set in the Document object.

[0049] The embodiment of the present application obtains a template URL and requests a first HTML tag, wherein the template URL is usually preset and points to a web page containing a standard HTML structure, and the structure of this web page is the target of parsing and imitation by the embodiment of the present application. In actual applications, the template URL may be stored in a configuration file, or may be dynamically obtained through other means (such as user input, database query, etc.).

[0050] A request is sent to the template URL through an HTTP request (such as a GET request) to obtain the HTML content of the web page. After the server responds to the request, the HTML content is returned.

[0051] The client (i.e., the processing program of the embodiment of the present application) receives these HTML contents and processes them as the "first HTML tag" for subsequent processing. The "first HTML tag" here actually refers to the beginning of the entire HTML document, but for ease of understanding, the embodiment of the present application regards it as a whole or a collection.

[0052] The received HTML content is parsed using an HTML parser (such as Jsoup, BeautifulSoup, lxml, etc., depending on the programming language used). The parsing process includes identifying HTML tags, attributes, text content, etc., and organizing them into a tree structure that is easy to operate.

[0053] The parser usually encapsulates the parsed HTML content into a Document object (or similar representation). The Document object is a high-level abstraction that provides methods to access and manipulate the content of an HTML document. Extract all Elements (i.e. HTML elements) from the Document object. The Elements object collection contains all the tag elements in the document, which can be any type of HTML tag (such as div, span, a, etc.).

[0054] Step 102, remove the non-parsed tags in the Elements object set to obtain a filtered Elements object set, where the non-parsed tags include link, script, and style tags.

[0055] The purpose of this step is to filter out those tags that are meaningful for structured data processing from the original set of HTML elements, while excluding those tags that do not contribute to structured information extraction or may interfere with the processing process.

[0056] First, it is necessary to clarify which tags are considered as non-parsed tags. In this embodiment, non-parsed tags include <link> , <script>和<style>。这些标签通常用于定义网页的样式、脚本和外部资源链接，它们的内容对于网页的结构化数据提取来说通常是不必要的，甚至可能是干扰因素。

[0057] 接下来，需要遍历原始的Elements对象集合，这个集合包含了从HTML文档中提取的所有元素。在遍历过程中，会对每个元素进行检查，以确定它是否属于非解析标签。

[0058] 对于每个被检查到的元素，如果它属于非解析标签之一，那么它就会被从Elements对象集合中移除。这样，经过遍历和过滤后，剩下的就是那些对结构化数据处理有意义的标签，即过滤后的Elements对象集合。

[0059] 在过滤过程中，需要特别注意保持元素之间的结构和关系不变。例如，如果一个元素内部包含了一个<script>标签，那么在移除<script>标签时，需要确保元素和其他子元素之间的层级关系不受影响。

[0060] 由于网页中可能包含大量的元素，因此这一步骤的处理性能对于整个网页内容结构化处理的效率来说至关重要。在实际应用中，可以通过使用高效的算法和数据结构来优化处理性能，例如使用哈希表来快速查找和移除非解析标签。

[0061] 通过这一步骤，可以确保后续的结构化数据处理过程更加准确和高效。过滤后的Elements对象集合将作为后续步骤的输入，用于提取网页中的结构化信息，并最终保存到数据库中供后续使用和分析。

[0062] 步骤103，对过滤后的Elements对象集合进行循环，并为过滤后的Elements对象集合中的每一个对象设置CurrentId值，CurrentId值为每一个对象对应循环的序号。

[0063] 在提取了Document对象中的Elements对象集合，并去除了非解析标签（如link、script和style标签）之后，需要对剩余的Elements对象进行遍历。在遍历的过程中，为每一个Elements对象分配一个唯一的标识符，即CurrentId值。这个值是根据对象在集合中的循环序号来确定的，因此它保证了每个对象都有一个独一无二的ID。

[0064] 在开始遍历Elements对象集合之前，我们首先需要初始化CurrentId值。通常，这个值可以从1开始，或者根据实际需求从其他数字开始。接下来，使用循环结构（如for循环或while循环）来遍历整个Elements对象集合。

[0065] 在每次循环中，将CurrentId值分配给当前遍历到的Elements对象。这个值在整个处理过程中是唯一的，并且与对象的顺序相关联。在分配完CurrentId值之后，需要更新CurrentId的值，以便在下一次循环中为下一个Elements对象分配一个新的、唯一的ID。

[0066] CurrentId值作为每个Elements对象的唯一标识符，使得本申请实施例可以在后续的处理过程中轻松地引用和识别这些对象。

[0067] 在构建树形结构对象时，CurrentId值也起到了关键作用。通过为每个标签分配一个唯一的ID，并设置其父标签的ID（即parentId属性），可以清晰地表示出标签之间的层级关系。在解析树形结构对象并生成节点数据时，CurrentId值也用于对节点数据进行筛选和命名，有助于更准确地提取和保存网页中的结构化信息。

[0068] 在设置CurrentId值时，必须确保每个对象的ID都是唯一的，可以通过在每次循环后更新CurrentId值来实现。如果在多线程或分布式环境中处理Elements对象集合，需要特别注意并发问题。在这种情况下，可能需要使用更复杂的机制来生成唯一的ID（如UUID）。

[0069] 综上所述，对过滤后的Elements对象集合进行循环，并为每个对象设置CurrentId值是网页内容结构化处理中的一个重要步骤。通过这一步骤，本申请实施例可以为每个Elements对象分配一个唯一的标识符，并在后续的处理过程中轻松地引用和识别这些对象。

[0070] 步骤104，定义Tag对象并组装List<tag>集合，构建树形结构对象。

[0071] Tag对象是一个用于表示HTML标签的数据结构，包含了以下关键属性：

[0072] id属性：表示标签的唯一标识符（ID）。在网页内容结构化处理过程中，每个标签都会被分配一个唯一的CurrentId值，这个值就是Tag对象的id属性值。

[0073] parentId属性：表示当前标签的父标签ID。如果当前标签有父标签，则parentId值等于父标签的CurrentId值；如果当前标签是顶级标签（即没有父标签），则parentId值为0。

[0074] name属性：表示标签的名称，如"div”、"span”、"a”等。这个属性用于标识标签的类型。

[0075] text属性：表示标签内的文本内容。对于某些标签（如"p”标签、"span”标签等），text属性可能包含重要的信息。

[0076] src属性：表示当前标签的src属性值。这个属性通常用于"img”标签、"script”标签等，用于指定资源文件的路径。如果当前标签没有src属性，则src属性值为Null。

[0077] href属性：表示当前标签的href属性值。这个属性通常用于"a”标签、"link”标签等，用于指定链接目标的URL。如果当前标签没有href属性，则href属性值为Null。

[0078] 在定义了Tag对象之后，需要将网页中的所有标签都转换为Tag对象，并将这些Tag对象存储在一个List<tag>集合中。这个集合包含了网页中所有标签的信息，是后续构建树形结构对象的基础。

[0079] 在组装了List<tag>集合之后，需要根据标签之间的父子关系来构建树形结构对象。这个树形结构对象能够直观地表示网页中标签的层级关系。

[0080] 根据List<tag>集合、id和parentId构建树形结构：遍历List<tag>集合中的每个Tag对象；对于每个Tag对象，检查其parentId属性；如果parentId为0，说明该Tag对象是顶级标签，将其添加到树形结构对象的根节点中；如果parentId不为0，说明该Tag对象有父标签，将其添加到其父标签的子节点列表中。

[0081] 在实际应用中，本申请实施例使用一个泛型为String的Tree类来表示树形结构对象中的每个节点。每个Tree对象都包含了一个值（通常是节点的标签名称）和一个子节点列表（List<Tree<String>>）。通过这种方式，可以构建一个完整的树形结构对象，用于表示网页中标签的层级关系。

[0082] 本申请实施例通过定义Tag对象并包含必要的属性（如id、parentId、name、text、src和href），可以准确地表示网页中的每个标签。然后，通过组装这些Tag对象到一个List<tag>集合中，并根据标签之间的父子关系构建树形结构对象，可以直观地表示网页中标签的层级关系。这为后续的网页内容解析和结构化处理提供了基础。

[0083] 步骤105，解析树形结构对象，生成树形结构对象数据，树形结构对象数据包括节点数据，对节点数据进行筛选和命名，形成模板数据和模板数据的属性名，将模板数据保存至模板数据库中。

[0084] 在网页内容结构化处理过程中，解析树形结构对象是一个关键步骤。这一步的目的是从HTML文档中提取出有用的信息，并将其组织成一个结构化的树形数据结构。这个树形结构对象包含了节点数据，每个节点都代表HTML文档中的一个元素。

[0085] 在解析过程中，会对每个节点数据进行筛选和命名，以形成模板数据和模板数据的属性名。这些模板数据随后会被保存至模板数据库中，以便后续使用。

[0086] 降低树形结构对象中的节点深度是一个优化步骤，旨在简化树形结构，提高处理效率。通过预设规则，可以去除那些不必要的节点，从而减少树形结构的复杂度。

[0087] 首先，本申请实施例判断树节点是否存在子节点。如果不存在子节点，则该节点为末端节点。接着，会检查末端节点的text、src和href属性是否同时为空。如果这三个属性都为空，那么该节点将被删除，并沿着层级结构向上追溯，迭代处理。

[0088] 从末端节点逆向溯源，当某个节点个数为1，并且包含的内容和父节点直接获取的纯文本内容相同时，该节点被视为冗余节点。此时，会去除该节点，使用其父节点来代替，并沿着层级结构向上追溯，迭代处理。

[0089] 另外，当某个节点及其同级节点的数量和每个节点的内容量满足预设公式时，也会将这些节点及其同级节点的内容进行整合，并入其父节点中。这样做可以减少节点的数量，进一步简化树形结构，预设公式如下所示：

[0090]

[0091] 其中，表示父节点的值，表示子节点的个数，表示预设阈值，

[0092] 表示父节点的值除以子节点的个数小于预设阈值，表示最大子节点的值，表示最小子节点的值；

[0093] 表示所有的子节点都不包括数字，对于每个节点，通过表示该节点包括数字，若，则包括数字，若，则不包括数字。

[0094] 在实际应用中，降低树形结构对象中的节点深度需要谨慎处理。因为过度简化可能会导致重要信息的丢失。因此，在设置预设规则时，需要充分考虑HTML文档的结构和内容特点，以确保解析结果的准确性和完整性。

[0095] 步骤106，解析同类站点内容，匹配模板数据，若无法匹配模板数据，则解析失败，并进行记录；根据模板数据的属性名匹配第二HTML标签，若第二HTML标签包括href属性或src属性，则判断href属性或src属性的值是文件还是外联，如果是文件，则根据第二HTML标签中的链接下载文件；如果是外联，则保存第二HTML标签中的链接。

[0096] 该步骤旨在确保从目标站点中提取的信息能够准确无误地映射到预定义的模板数据结构中。

[0097] 首先，本申请实施例会对同类站点的网页内容进行解析，涉及到发送HTTP请求以获取网页的HTML源代码，然后使用HTML解析器将源代码转换为可操作的Document对象或Elements对象集合。

[0098] 接着，会将这些解析后的内容与之前保存的模板数据进行匹配，匹配过程涉及比较标签名称、属性、层级结构等多个方面。如果解析后的内容与模板数据成功匹配，那么就可以根据模板数据的属性名来提取和结构化网页中的信息。匹配过程中，还可以使用各种算法和技术来提高匹配的准确性和效率，如正则表达式、XPath查询、CSS选择器等。

[0099] 如果无法匹配模板数据，则表明当前网页的结构或内容与预定义的模板存在差异。此时，系统会记录解析失败的情况，并可能触发一些后续操作，如通知管理员、尝试使用其他模板进行匹配等。

[0100] 在成功匹配模板数据后，会继续根据模板数据的属性名来匹配网页中的第二HTML标签。这些第二HTML标签包含了需要提取的具体信息，如链接、图片等。匹配过程可能涉及检查标签的名称、属性（如href、src等）以及属性值是否与模板数据中的属性名相匹配。

[0101] 对于匹配成功的第二HTML标签，特别关注其href和src属性（如果存在的话）。如果href或src属性的值指向一个文件，则根据该链接下载文件，并将其保存到指定的位置。如果href或src属性的值指向一个外联资源（如另一个网页、图片等），则保存该链接的URL，以便后续使用。

[0102] 在处理完所有匹配的第二HTML标签后，将提取到的信息（无论是文件还是链接）进行记录和存储。这些信息会被保存到数据库中，以供后续的分析、处理或展示使用。

[0103] 综上所述，该步骤涉及到多个复杂的过程和技术，包括HTML解析、模板匹配、属性名匹配以及文件或链接的处理等。通过这一过程，能够从网页中提取出有价值的信息，并将其结构化为易于理解和操作的数据格式。

[0104] 步骤107，将过滤后的Elements对象集合中每个对象的CurrentId值、树形结构对象数据以及解析同类站点内容时得到的文件或链接，结合模板数据的属性名，组装成结构化对象数据，并将结构化对象数据保存至采集数据库。

[0105] 在前面的步骤中，通过一系列的处理手段，如获取模板URL、解析HTML标签、提取Elements对象集合、去除非解析标签、设置CurrentId值、定义Tag对象并组装List<tag>集合以及构建树形结构对象等，成功地从网页中提取出了所需的数据，并为之分配了相应的属性名。

[0106] 这些数据包括网页中的文本内容、图片链接、视频链接等，而属性名则用于标识这些数据的类型和含义，如"标题”、"作者”、"发布时间”等。

[0107] 有了这些数据和属性名之后，下一步就是将它们组装成结构化对象数据。结构化对象数据是指按照一定规则和组织结构存储的数据，它使得数据更加清晰、易于理解和使用。

[0108] 具体来说，本申请实施例根据数据的类型和属性名，创建一个或多个数据结构（如类、结构体等），然后将解析出的数据填充到这些数据结构中。这样，每个数据结构就代表了一个结构化对象，它包含了网页中的一部分信息。

[0109] 最后，需要将这些结构化对象数据保存至采集数据库中。采集数据库是一个专门用于存储从网页中提取出的数据的数据库，它可以是关系型数据库（如MySQL、PostgreSQL等），也可以是非关系型数据库（如MongoDB、Cassandra等）。

[0110] 在保存数据时，需要确保数据的完整性和准确性。这可以通过在数据库中设置适当的约束条件（如主键约束、外键约束等）和验证规则（如数据格式验证、数据范围验证等）来实现。

[0111] 此外，为了提高数据的查询效率和可维护性，还需要对数据库进行适当的设计和优化。这包括选择合适的存储引擎、设置合理的索引、优化查询语句等。

[0112] 如图2所示，本申请实施例提供了一种网页内容结构化处理装置，包括元素获取模块201、过滤循环模块202、树形结构模块203、解析处理模块204和结构化模块205，其中：

[0113] 元素获取模块201，被配置为用于获取模板URL，请求模板URL的URL地址，得到第一HTML标签；解析第一HTML标签，将第一HTML标签转化为Document对象，提取Document对象中的Elements对象集合；

[0114] 过滤循环模块202，被配置为用于去除Elements对象集合中的非解析标签，得到过滤后的Elements对象集合，非解析标签包括link、script和style标签；对过滤后的Elements对象集合进行循环，并为过滤后的Elements对象集合中的每一个对象设置CurrentId值，CurrentId值为每一个对象对应循环的序号；

[0115] 树形结构模块203，被配置为用于定义Tag对象并组装List<tag>集合，构建树形结构对象；解析树形结构对象，生成树形结构对象数据，树形结构对象数据包括节点数据，对节点数据进行筛选和命名，形成模板数据和模板数据的属性名，将模板数据保存至模板数据库中；

[0116] 解析处理模块204，被配置为用于解析同类站点内容，匹配模板数据，若无法匹配模板数据，则解析失败，并进行记录；根据模板数据的属性名匹配第二HTML标签，若第二HTML标签包括href属性或src属性，则判断href属性或src属性的值是文件还是外联，如果是文件，则根据第二HTML标签中的链接下载文件；如果是外联，则保存第二HTML标签中的链接；

[0117] 结构化模块205，被配置为用于将过滤后的Elements对象集合中每个对象的CurrentId值、树形结构对象数据以及解析同类站点内容时得到的文件或链接，结合模板数据的属性名，组装成结构化对象数据，并将结构化对象数据保存至采集数据库。

[0118] 本申请实施例的网页内容结构化处理装置可以是计算机设备，也可以是计算机设备中的部件，例如集成电路或芯片。该计算机设备可以是终端，也可以为除终端之外的其他设备。示例性的，计算机设备可以为手机、平板电脑、笔记本电脑、掌上电脑、车载计算机设备、移动上网装置(Mobile Internet Device，MID)、超级移动个人计算机(Ultra-MobilePersonal Computer，UMPC)、上网本或者个人数字助理(Personal Digital Assistant，PDA)等，还可以为服务器、网络附属存储器(Network Attached Storage，NAS)、个人计算机(Personal Computer，PC)等，本申请实施例不作具体限定。

[0119] 本申请实施例提供的网页内容结构化处理装置能够实现图1的网页内容结构化处理方法实施例实现的各个过程，为避免重复，这里不再赘述。

[0120] 本申请实施例还提供一种计算机设备，如图3所示，该计算机设备包括处理器301和存储器302，存储器302上存储有可在处理器301上运行的程序或指令，该程序或指令被处理器301执行时实现上述网页内容结构化处理方法的各个步骤，且能达到相同的技术效果，为避免重复，这里不再赘述。

[0121] 需要说明的是，本申请实施例的计算机设备包括上述的移动计算机设备和非移动计算机设备。

[0122] 存储器302可用于存储软件程序以及各种数据。存储器302可主要包括存储程序或指令的第一存储区和存储数据的第二存储区，其中，第一存储区可存储操作系统、至少一个功能所需的应用程序或指令(比如声音播放功能、图像播放功能等)等。此外，存储器302可以包括易失性存储器或非易失性存储器，或者，存储器302可以包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EPROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous DRAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SDRAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SDRAM，ESDRAM)、同步连接动态随机存取存储器(Synch link DRAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本申请实施例的存储器302包括但不限于这些和任意其它适合类型的存储器。

[0123] 处理器301可包括一个或多个处理单元；可选的，处理器301集成应用处理器和调制解调处理器，其中，应用处理器主要处理涉及操作系统、用户界面和应用程序等的操作，调制解调处理器主要处理无线通信信号，如基带处理器。可以理解的是，上述调制解调处理器也可以不集成到处理器301中。

[0124] 本申请实施例还提供一种可读存储介质，可读存储介质上存储有程序或指令，该程序或指令被处理器执行时实现上述网页内容结构化处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。

[0125] 本申请实施例还提供了一种芯片，芯片包括处理器和通信接口，通信接口和处理器耦合，处理器用于运行程序或指令，实现上述网页内容结构化处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。

[0126] 应理解，本申请实施例提到的芯片还可以称为系统级芯片、系统芯片、芯片系统或片上系统芯片等。

[0127] 本申请实施例还提供一种计算机程序产品，该程序产品被存储在存储介质中，该程序产品被至少一个处理器执行以实现如上述网页内容结构化处理方法实施例的各个过程，且能达到相同的技术效果，为避免重复，这里不再赘述。

[0128] 需要说明的是，在本申请中，术语"包括”、"包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者装置不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者装置所固有的要素。在没有更多限制的情况下，由语句"包括一个……”限定的要素，并不排除在包括该要素的过程、方法、物品或者装置中还存在另外的相同要素。此外，需要指出的是，本申请实施方式中的方法和装置的范围不限按示出或讨论的顺序来执行功能，还可包括根据所涉及的功能按基本同时的方式或按相反的顺序来执行功能，例如，可以按不同于所描述的次序来执行所描述的方法，并且还可以添加、省去、或组合各种步骤。另外，参照某些示例所描述的特征可在其他示例中被组合。

[0129] 上面结合附图对本申请的实施例进行了描述，但是本申请并不局限于上述的具体实施方式，上述的具体实施方式仅仅是示意性的，而不是限制性的，本领域的普通技术人员在本申请的启示下，在不脱离本申请宗旨和权利要求所保护的范围情况下，还可做出很多形式，均属于本申请的保护之内。< / script> < / tag> < / string> < / tag> < / tag> < / string> < / tag> < / tag>

Claims

1. A method for structuring webpage content, characterized in that: The method comprises: Obtaining a template URL, requesting the URL address of the template URL, and obtaining a first HTML tag; parsing the first HTML tag, converting the first HTML tag into a Document object, and extracting an Elements object set in the Document object; Removing non-parsed tags in the Elements object set to obtain a filtered Elements object set, wherein the non-parsed tags include link, script, and style tags; Loop the filtered Elements object set, and set a CurrentId value for each object in the filtered Elements object set, where the CurrentId value is a sequence number corresponding to the loop of each object; Define Tag objects and populate List <tag>A tree structure object is constructed by a collection, wherein the Tag object includes id, parentId, name, text, src, and href attributes, wherein id represents the tag id, and the id value is equal to the CurrentId value; parentId represents the parent tag id, and the parentId value is equal to the CurrentId value of the parent tag. If the parent tag does not exist, the parentId value is 0; name represents the tag name; src represents the src attribute value of the current tag. If the attribute does not exist, the src attribute value is Null; href represents the href attribute value of the current tag. If the attribute does not exist, the href attribute value is Null; according to the List <tag>The collection, id and parentId are constructed as List <Tree <string> >Tree structure object;< / string> < / tag> < / tag> Parsing the tree structure object to generate tree structure object data, the tree structure object data including node data, screening and naming the node data to form template data and attribute names of the template data, and saving the template data into a template database; Parse the content of the same site and match the template data. If the template data cannot be matched, the parsing fails and is recorded. Match the second HTML tag according to the attribute name of the template data. If the second HTML tag includes an href attribute or a src attribute, determine whether the value of the href attribute or the src attribute is a file or an external link. If it is a file, download the file according to the link in the second HTML tag. If it is an external link, save the link in the second HTML tag. The CurrentId value of each object in the filtered Elements object set, the tree structure object data, and the files or links obtained when parsing the content of similar sites are combined with the attribute name of the template data to assemble into structured object data, and the structured object data is saved to the collection database.

2. The webpage content structuring processing method according to claim 1, characterized in that: The Tag object includes an id attribute, a parentId attribute, a name attribute, a text attribute, a src attribute, and a href attribute.

3. The webpage content structuring processing method according to claim 1, characterized in that: Before parsing the tree structure object, the method further includes reducing the node depth in the tree structure object according to a preset rule.

4. The webpage content structuring processing method according to claim 3, characterized in that: The tree structure object includes tree nodes, and reducing the node depth in the tree structure object by using a preset rule includes: Determine whether the tree node has child nodes. If no child nodes exist, it is the terminal node. When the text, src and href of the terminal node are empty at the same time, delete the node object and trace back upward along the hierarchical structure for iterative processing.

5. The webpage content structuring processing method according to claim 4, characterized in that: Tracing back from the end node, when the number of the node is 1 and the content contained is the same as the plain text content directly obtained from the parent node, remove the node, use its parent node, and trace back upward along the hierarchical structure for iterative processing; When the number of the node and its sibling nodes and the content of each node satisfy the following formula: ； in, Represents the value of the parent node. Indicates the number of child nodes. Indicates the preset threshold value, Indicates that the value of the parent node divided by the number of child nodes is less than the preset threshold. Represents the value of the largest child node, Indicates the value of the smallest child node; Indicates that all child nodes do not include numbers. For each node ,pass Indicates that the node contains numbers. If ,but Including numbers, if ,but Do not include numbers; Integrate the content of this node and its sibling nodes into its parent node, use the original content of its parent node, and trace back upward along the hierarchical structure for iterative processing.

6. A web page content structuring processing device, characterized in that: The device comprises: The element acquisition module is configured to acquire a template URL, request a URL address of the template URL, and obtain a first HTML tag; parse the first HTML tag, convert the first HTML tag into a Document object, and extract an Elements object set in the Document object; A filtering loop module is configured to remove non-parsed tags in the Elements object set to obtain a filtered Elements object set, wherein the non-parsed tags include link, script and style tags; loop the filtered Elements object set, and set a CurrentId value for each object in the filtered Elements object set, wherein the CurrentId value is a sequence number corresponding to the loop of each object; The tree structure module is configured to define Tag objects and assemble List <tag>A tree structure object is constructed by a collection, wherein the Tag object includes id, parentId, name, text, src, and href attributes, wherein id represents the tag id, and the id value is equal to the CurrentId value; parentId represents the parent tag id, and the parentId value is equal to the CurrentId value of the parent tag. If the parent tag does not exist, the parentId value is 0; name represents the tag name; src represents the src attribute value of the current tag. If the attribute does not exist, the src attribute value is Null; href represents the href attribute value of the current tag. If the attribute does not exist, the href attribute value is Null; according to the List <tag>The collection, id and parentId are constructed as List <Tree <string> >Tree structure object; parsing the tree structure object to generate tree structure object data, the tree structure object data including node data, screening and naming the node data to form template data and attribute names of the template data, and saving the template data into a template database;< / string> < / tag> < / tag> The parsing processing module is configured to parse the content of the same site and match the template data. If the template data cannot be matched, the parsing fails and is recorded; match the second HTML tag according to the attribute name of the template data, if the second HTML tag includes an href attribute or a src attribute, determine whether the value of the href attribute or the src attribute is a file or an external link, if it is a file, download the file according to the link in the second HTML tag; if it is an external link, save the link in the second HTML tag; The structuring module is configured to assemble the CurrentId value of each object in the filtered Elements object set, the tree structure object data, and the files or links obtained when parsing the content of the same site into structured object data in combination with the attribute name of the template data, and save the structured object data to the collection database.

7. A webpage content structuring processing system, the system comprising a processor and a memory, wherein the memory stores a computer program, characterized in that: The computer program is loaded and executed by the processor to implement the web page content structuring processing method according to any one of claims 1 to 5.

8. A computer-readable storage medium, wherein a computer program is stored in the storage medium, characterized in that: When the computer program is executed by a processor, it is used to implement the web page content structuring processing method according to any one of claims 1 to 5.

Images