Text extraction method, device, electronic device and storage medium

Abstract

The present application discloses a text extraction method, device, electronic device and storage medium, and relates to the field of data collection. The method includes: determining at least one target DOM tree based on the web page tags in the web page source code, wherein each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is a non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for circling the web page body; according to the text weight and structure weight of each target tree node, the body tree node is screened from all the target tree nodes; and the text block corresponding to the body tree node is extracted from the web page. The present application solves the technical problem of inaccurate extraction of web page body content caused by extracting the text content of the web page based on statistics in the prior art.

Description

Technical Field

[0001] The present application relates to the field of data collection, and in particular to a text extraction method, device, electronic device and storage medium. Background Art

[0002] Web page text extraction is a key task in the field of data collection. Its core purpose is to accurately separate the information content required by users from diverse network resources. In the era of Internet information explosion, web page content is becoming more and more abundant, but it is also accompanied by a large amount of noise information, such as navigation bars, advertisements, copyright notices and user comments, which are often not the focus of users. With the development of web page technology, especially the increase of dynamic content, traditional web page text extraction methods are facing more and more challenges.

[0003] The web page text extraction method provided in the related art is mainly based on extracting the text content of the web page in a statistical manner. For example, the text content of the web page is determined by counting the frequency of occurrence of words or punctuation marks. Although this method takes into account the characteristics of text and punctuation, due to the differences between different languages ​​and the diversity of website content, the accuracy of its text extraction is not ideal.

[0004] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0005] The embodiments of the present application provide a text extraction method, device, electronic device and storage medium to at least solve the technical problem of inaccurate extraction of web page text content caused by extracting the text content of a web page in a statistical manner in the prior art.

[0006] According to one aspect of an embodiment of the present application, a text extraction method is provided, comprising: determining at least one target DOM tree based on web page tags in a web page source code, wherein tree nodes in each target DOM tree are target tree nodes, text blocks corresponding to the target tree nodes are non-empty texts, and web page tags corresponding to at least one layer of upper-level nodes of the target tree nodes are target web page tags, and the target web page tags include target elements for circumscribing the body of the web page; filtering out body tree nodes from all target tree nodes based on text weights and structural weights of each target tree node, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structural weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page; and extracting text blocks corresponding to the body tree nodes from the web page.

[0007] In some embodiments, in the text extraction method provided by the present application, at least one target DOM tree is determined based on the web page tags in the web page source code, including: generating an initial DOM tree based on the web page source code, wherein each web page tag in the web page source code corresponds to a tree node in the initial DOM tree; filtering non-critical web page tags in the web page source code, and deleting the tree nodes corresponding to the non-critical web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that the text block corresponding to the non-critical web page tag belongs to the web page body is lower than a preset probability; detecting the tree nodes in the first DOM tree that belong to the target tree nodes, and determining the target DOM tree based on the detection results.

[0008] In some embodiments, in the text extraction method provided by the present application, tree nodes belonging to the target tree node in the first DOM tree are detected, and the target DOM tree is determined according to the detection result, including: determining L tag DOM trees according to the L tree nodes in the first DOM tree through a breadth search strategy and a depth search strategy, wherein L is an integer greater than or equal to 1, the root node of each tag DOM tree is a tree node of the first DOM tree, the breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree; detecting whether each tree node in each tag DOM tree belongs to the target tree node, and according to the detection result, deleting the tree nodes that do not belong to the target tree node in each tag DOM tree to obtain the target DOM tree.

[0009] In some embodiments, in the text extraction method provided by the present application, before filtering out the text tree node from all the target tree nodes according to the text weight and the structural weight of each target tree node, the text extraction method further includes: for the i-th target tree node in any target DOM tree, determining the target DOM tree with the i-th target tree node as the root node as the second DOM tree, where i is an integer greater than or equal to 1; obtaining the text length of the text block corresponding to each tree node in the second DOM tree; using the hierarchical information of each tree node in the second DOM tree in the second DOM tree as the node depth of the tree node; determining the structural weight of each tree node in the second DOM tree according to the node depth of each tree node in the second DOM tree and the text length of the corresponding text block; summing the structural weights of all tree nodes in the second DOM tree, and using the summation result as the structural weight of the i-th target tree node.

[0010] In some embodiments, in the text extraction method provided by the present application, before filtering out the main text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, the text extraction method further includes: for the i-th target tree node in any target DOM tree, determining a tag DOM tree with the i-th target tree node as a root node as a third DOM tree, wherein the tag DOM tree includes target tree nodes and non-target tree nodes, and i is an integer greater than or equal to 1; obtaining the number of target tree nodes and the number of non-target tree nodes in the third DOM tree; determining the text weight of the i-th target tree node according to the number of target tree nodes and the number of non-target tree nodes in the third DOM tree and a preset text weight hyperparameter.

[0011] In some embodiments, in the text extraction method provided by the present application, the main text tree nodes are screened from all the target tree nodes according to the text weight and structure weight of each target tree node, including: obtaining preset hyperparameters of the text weight; calculating the product of the text weight of each target tree node and the preset hyperparameter to obtain the target text weight of each target tree node; calculating the product of the target text weight and the structure weight of each target tree node to obtain the target weight of each target tree node; according to the target weight of each target tree node, the main text tree nodes are screened from all the target tree nodes.

[0012] In some embodiments, in the text extraction method provided by the present application, according to the target weight of each target tree node, the main text tree node is screened from all the target tree nodes, including: according to the target weight of each target tree node, all the target tree nodes are sorted, and the main text tree nodes are screened from all the target tree nodes according to the sorting result; or, according to the target weight of each target tree node, the target tree nodes whose target weights are greater than a preset threshold are screened from all the target tree nodes as the main text tree nodes.

[0013] According to another aspect of the present application, a text extraction device is also provided, which includes: a first processing unit, which is used to determine at least one target DOM tree based on the web page tags in the web page source code, wherein the tree nodes in each target DOM tree are target tree nodes, the text block corresponding to the target tree node is non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for circling the web page text; a second processing unit, which is used to filter out the text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page; a third processing unit, which is used to extract the text block corresponding to the text tree node from the web page.

[0014] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the text extraction method provided in any one of the above-mentioned embodiments.

[0015] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the text extraction method provided in any one of the above-mentioned embodiments.

[0016] In the present application, at least one target DOM tree is first determined based on the web page tags in the web page source code, wherein each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is a non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for delineating the web page body. Then, based on the text weight and structure weight of each target tree node, the body tree node is screened from all the target tree nodes, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page. Finally, the text block corresponding to the body tree node is extracted from the web page.

[0017] As can be seen from the above content, a node screening method for a DOM tree is provided, including: identifying a tree node whose corresponding text block is non-empty text and whose web page tag corresponding to at least one layer of upper-level nodes is a target web page tag (a target element that can circle out the body of the web page, that is, a web page tag with body meaning) as a target tree node, and then calculating the text weight and the structure weight for each target tree node, so as to comprehensively determine which target tree nodes belong to the body tree node in terms of the two dimensions of text information and structure information, and finally extracting the text block corresponding to the body tree node from the web page. Compared with the statistical method of extracting the text content of the web page in the prior art, the present application comprehensively determines the body information from at least the following four dimensions: the first dimension, whether the text block corresponding to the DOM tree node is empty text; the second dimension, whether the web page tag corresponding to at least one layer of upper-level nodes (such as the parent node and the grandparent node) of the DOM tree node is the target web page tag; the third dimension, the text weight of the DOM tree node; and the fourth dimension, the structure weight of the DOM tree node. By providing more reference dimensions, the present application can improve the accuracy of extracting the body content of a web page, thereby solving the technical problem of inaccurate extraction of the body content of a web page caused by extracting the text content of a web page in a statistical manner in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0019] Figure 1 is a flowchart of an optional text extraction method according to an embodiment of the present application;

[0020] Figure 2 is a schematic diagram of an optional first DOM tree according to an embodiment of the present application;

[0021] Figure 3 is a schematic diagram of an optional process of determining a target DOM tree according to an embodiment of the present application;

[0022] Figure 4 is a flowchart of another optional text extraction method according to an embodiment of the present application;

[0023] Figure 5 It is a schematic diagram of an optional text extraction device according to an embodiment of the present application. DETAILED DESCRIPTION

[0024] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0026] It should also be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) collected by this application are information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data are in compliance with the relevant laws, regulations and standards of the relevant regions, necessary confidentiality measures are taken, and public order and good customs are not violated, and corresponding operation portals are provided for users to choose to authorize or refuse. For example, an interface is set up between this system and relevant users or institutions. Before obtaining relevant information, it is necessary to send an acquisition request to the aforementioned user or institution through the interface, and obtain relevant information after receiving the consent information fed back by the aforementioned user or institution.

[0027] In an optional embodiment, a text extraction system can be used as the execution subject of the text extraction method in the embodiment of the present application, wherein the text extraction system can be a software system or an embedded system combining software and hardware. Of course, in addition to using the text extraction system as the execution subject, other forms of devices and equipment can also be used as the execution subject of the technical solution of the embodiment of the present application, and the present application does not specifically limit the specific type of the execution subject.

[0028] For the convenience of description, the technical solution of the embodiment of the present application is explained below using the text extraction system as the execution entity.

[0029] According to an embodiment of the present application, an embodiment of a text extraction method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0030] Figure 1 is a flowchart of an optional text extraction method according to an embodiment of the present application, such as Figure 1 As shown, the method comprises the following steps:

[0031] Step S101, determining at least one target DOM tree according to web page tags in the web page source code.

[0032] In step S101, each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for circumscribing the web page text.

[0033] In some examples, at least one level of upper-level node of the target tree node may be a “parent node of the target tree node”.

[0034] In some other examples, at least one level of superior nodes of the target tree node may also be “the parent node and grandparent node of the target tree node”.

[0035] In some other examples, at least one level of superior nodes of the target tree node may also be “the parent node, the grandparent node, and the great-grandparent node of the target tree node”.

[0036] Those skilled in the art will appreciate that, depending on the actual application scenario, at least one layer of parent nodes of the target tree node may be materialized as parent nodes of various levels including the target tree node, and the present application does not impose any particular limitation on the specific number of layers of parent nodes.

[0037] However, it should be noted that the more specific layers of parent nodes there are, the more stringent the target tree node screening conditions are. Correspondingly, the number of target tree nodes finally determined may be smaller, but the target DOM tree that can be determined in the end can also correspond to the web page body more accurately.

[0038] In the embodiment of the present application, the DOM tree (Document Object Model Tree) refers to a tree model used to represent the structure of a web page document. The DOM tree can organize the elements in an HTML (HyperText Markup Language) document into a hierarchical data structure so that it can be accessed and operated by a program. In the DOM tree, each node represents an element in the document, and the nodes are connected through a parent-child relationship to form a tree structure.

[0039] The main features of the DOM tree include:

[0040] 1. Hierarchy: The elements in the DOM tree are organized according to their nesting relationship in the document to form a hierarchical structure.

[0041] 2. Node: Each element in the DOM tree is a node, and a node can contain child nodes, parent nodes, and sibling nodes.

[0042] 3. Attributes and methods: Nodes in the DOM tree have attributes and methods that can be used to access and manipulate the content, style, and events of the elements.

[0043] 4. Compatibility: DOM tree is a general data structure that can be used to represent various types of documents, such as HTML, XML (Extensible Markup Language), etc.

[0044] By manipulating the DOM tree, developers can implement functions such as dynamic modification of web page content, style adjustment, and event processing.

[0045] In some embodiments, the text extraction system can obtain an initial DOM tree based on the web page source code conversion, wherein a tree node in the initial DOM tree corresponds to a web page tag in the web page source code. Then, the text extraction system can filter the web page tags in the web page source code, including: filtering non-key tags and meaningless tags in the web page source code, wherein the probability that the text block corresponding to the non-key web page tag belongs to the web page body is lower than a preset probability, and the meaningless tag does not include a target element for delineating the web page body.

[0046] In some embodiments, after the filtering operation is completed, only the target web page tag of the web page source code is retained. On this basis, all tree nodes in the initial DOM tree except the target tree node corresponding to the target web page tag are deleted, and a target DOM tree consisting only of target tree nodes can be obtained.

[0047] In some embodiments, the target web page tag is also referred to as a web page tag with text meaning. For example, such web page tags include target elements such as article, body, and main.

[0048] In some embodiments, web page tags that do not have text meaning do not include the above-mentioned target elements, and web page tags that do not have text meaning also include elements such as comment, combx, disqus, foot, header, etc.

[0049] Step S102, according to the text weight and structure weight of each target tree node, the text tree node is screened from all the target tree nodes.

[0050] In step S102, the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page.

[0051] In some embodiments, after obtaining the target DOM tree, the text weight and structural weight of each target tree node in the target DOM tree may be calculated, and then based on the text weight and structural weight of each target tree node, the text tree nodes may be screened from all the target tree nodes to obtain the main text tree nodes. For example, the text weight and structural weight of each target tree node may be weighted and summed to obtain the comprehensive weight of each target tree node, and then the first N target tree nodes with larger comprehensive weights may be selected as the main text tree nodes.

[0052] Step S103, extracting the text block corresponding to the text tree node from the web page.

[0053] In some embodiments, after obtaining the text tree node, the text extraction system can extract the text block corresponding to the text tree node from the web page to form the target text, that is, determine the target text to be the text content of the web page.

[0054] Based on the contents of the above steps S101 to S103, it can be known that in the present application, at least one target DOM tree is first determined based on the web page tags in the web page source code, wherein each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is a non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for delineating the body of the web page. Then, according to the text weight and structural weight of each target tree node, the body tree node is screened from all the target tree nodes, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structural weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page. Finally, the text block corresponding to the body tree node is extracted from the web page.

[0055] As can be seen from the above content, a node screening method for a DOM tree is provided, including: identifying a tree node whose corresponding text block is non-empty text and whose web page tag corresponding to the parent node is a target web page tag (a meaningful tag) as a target tree node, and then calculating the text weight and structure weight for each target tree node, thereby comprehensively determining which target tree nodes belong to the text tree node in terms of text information and structure information, and finally extracting the text block corresponding to the text tree node from the web page. Compared with the statistical method for extracting the text content of a web page in the prior art, the present application comprehensively determines the text information from at least the following four dimensions:

[0056] The first dimension is whether the text block corresponding to the DOM tree node is empty text;

[0057] The second dimension is whether the web page tag corresponding to at least one layer of upper-level nodes (such as parent nodes and grandparent nodes) of the DOM tree node is the target web page tag;

[0058] The third dimension is the text weight of the DOM tree node;

[0059] The fourth dimension is the structural weight of the DOM tree nodes.

[0060] By providing more reference dimensions, the present application can improve the accuracy of extracting the body content of a web page, thereby solving the technical problem of inaccurate extraction of the body content of a web page caused by extracting the text content of a web page in a statistical manner in the prior art.

[0061] In an optional embodiment, determining a target DOM tree based on web page tags in a web page source code includes: generating an initial DOM tree based on the web page source code, wherein each web page tag in the web page source code corresponds to a tree node in the initial DOM tree; filtering non-critical web page tags in the web page source code, and deleting tree nodes corresponding to the non-critical web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that a text block corresponding to the non-critical web page tag belongs to the web page body is lower than a preset probability; detecting tree nodes in the first DOM tree that belong to the target tree nodes, and determining the target DOM tree based on the detection result.

[0062] In some embodiments, the text extraction system can first obtain the web page HTML source code (i.e., the web page source code mentioned above) from the real network environment, and generate an initial DOM tree based on the web page HTML source code. It should be noted that each web page tag in the web page HTML source code corresponds to a tree node of the initial DOM tree.

[0063] In some embodiments, the text extraction system can also clean the web page tags in the HTML source code of the web page, and remove irrelevant tags (i.e., non-critical web page tags), where non-critical web page tags include: HTML, HEAD, META, TITLE, STYLE, LINK, OPTION, LABEL, IFRAME, UL, OL, LI, DD, DL, DT, H1, H2, H3 and other tags.

[0064] In some embodiments, after filtering non-key web page tags in the HTML source code of the web page, the text extraction system also needs to delete the tree nodes corresponding to the non-key web page tags in the initial DOM tree, and then obtain the first DOM tree.

[0065] It should be noted that, in actual applications, other web page tags can be newly added as non-critical web page tags. The judgment principle for non-critical web page tags in this application is mainly: if the probability that the text block corresponding to a web page tag belongs to the web page body is lower than the preset probability, then the web page tag is determined as a non-critical web page tag, where the preset probability can be customized.

[0066] It should also be noted that the tree nodes in the first DOM tree can be regarded as candidate tree nodes, and the web page tags corresponding to the candidate tree nodes can be regarded as candidate tags.

[0067] From the above content, it can be seen that in the web page text extraction method proposed in this application, the HTML source code of the web page is collected, and then the collected HTML of the web page is preprocessed to remove irrelevant tags, such as tags representing fonts, headers, tables, and lists, and all tags and DOM trees in the HTML source code of the web page that may represent the main content of the text are extracted as candidates.

[0068] In an optional embodiment, detecting tree nodes in the first DOM tree that belong to the target tree node, and determining the target DOM tree according to the detection result, includes: determining L tag DOM trees according to the L tree nodes in the first DOM tree through a breadth search strategy and a depth search strategy, wherein L is an integer greater than or equal to 1, the root node of each tag DOM tree is a tree node of the first DOM tree, the breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree; detecting whether each tree node in each tag DOM tree belongs to the target tree node, and according to the detection result, deleting the tree nodes that do not belong to the target tree node in each tag DOM tree to obtain the target DOM tree.

[0069] In some embodiments, the present application provides two node search strategies, namely, a breadth search strategy and a depth search strategy. According to the two node search strategies, L tag DOM trees are first determined based on the L tree nodes in the first DOM tree. The breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree.

[0070] For example, Figure 2 is a schematic diagram of an optional first DOM tree according to an embodiment of the present application. Assuming that the depth search strategy stipulates that the maximum depth of a tag DOM tree is 2, and the breadth search strategy stipulates that the maximum breadth of a tag DOM tree is not limited, then based on Figure 2 From the six tree nodes A, B, C, D, E, and F in the first DOM tree in , six tag DOM trees can be obtained, including: the tag DOM tree corresponding to the tree node "A" is a DOM tree composed of the three tree nodes A, B, and C (the depth is 2, the breadth is not limited, and all brother nodes at the same level are included); the tag DOM tree corresponding to the tree node "B" is a DOM tree composed of the two tree nodes B and D; the tag DOM tree corresponding to the tree node "C" is a DOM tree composed of the three tree nodes C, E, and F; the tag DOM tree corresponding to the tree node "D" only has the tree node "D" itself; the tag DOM tree corresponding to the tree node "E" only has the tree node "E" itself; the tag DOM tree corresponding to the tree node "F" only has the tree node "F" itself.

[0071] In some embodiments, according to the deep search strategy of the present application, for each tag DOM tree, it is also possible to detect whether the text block of each tree node in the tag DOM tree is empty text, and determine whether the parent node and grandparent node of each tree node in the tag DOM tree are meaningless non-text tags. If the text block of a tree node is not empty text, and the parent node and grandparent node tags of the tree node belong to the target web page tag with text meaning, the tree node is retained as the target tree node.

[0072] In some embodiments, according to the breadth search strategy of the present application, all brother nodes of each tree node in the tag DOM tree can also be traversed, and the above judgment is performed in the same way. If the text block of a tree node is not empty text, and the tags of the parent node and grandparent node of the tree node belong to the target web page tag with text meaning, then the tree node is retained as the target tree node; if the text block of a tree node is empty text, then the tree node and the web page tag corresponding to the tree node are deleted; if the web page tag of the parent node or grandparent node of a tree node is a non-target web page tag, then the tree node and the web page tag corresponding to the tree node are deleted.

[0073] Through the above process, the tree nodes that do not belong to the target tree node in each tag DOM tree are all deleted, and a target DOM tree that only retains the target tree node is obtained.

[0074] In some embodiments, for each candidate tag's tag DOM tree, a breadth search strategy and a depth search strategy are used to filter out tree nodes corresponding to empty text and tree nodes corresponding to meaningless tags in the tag DOM tree, and then obtain the target DOM tree.

[0075] It should be noted that, based on each candidate tag that may represent the main content and the tag DOM tree corresponding to the candidate tag, the tag DOM tree is traversed by using a depth search and a breadth search, and text detection and tag detection are further performed on upper nodes such as parent nodes and grandparent nodes, as well as brother nodes, to filter out tree nodes with empty text content and tree nodes whose web page tags of parent nodes or grandparent nodes are non-target web page tags, thereby achieving the purpose of retaining the target tree nodes with certain text content and meeting the main content identifier in each candidate tag DOM tree.

[0076] It should also be noted that since the target DOM tree is derived based on the tag DOM, while the breadth search strategy is used to determine the maximum breadth of the tag DOM tree and the depth search strategy is used to determine the maximum depth of the tag DOM tree, the breadth search strategy can also be used to determine the maximum breadth of the target DOM tree and the depth search strategy can also be used to determine the maximum depth of the target DOM tree.

[0077] In an optional embodiment, before filtering out the text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, the text extraction system can determine the target DOM tree with the i-th target tree node as the root node as the second DOM tree for the i-th target tree node in any target DOM tree, where i is an integer greater than or equal to 1. Then, the text length of the text block corresponding to each tree node in the second DOM tree is obtained, and the hierarchical information of each tree node in the second DOM tree in the second DOM tree is used as the node depth of the tree node. Subsequently, the structure weight of each tree node in the second DOM tree is determined according to the node depth of each tree node in the second DOM tree and the text length of the corresponding text block. Finally, the structure weights of all tree nodes in the second DOM tree are summed up and the sum result is used as the structure weight of the i-th target tree node.

[0078] In some embodiments, assuming that the tag corresponding to the i-th target tree node is the I-th tag in the web page source code, the target DOM tree with the i-th target tree node as the root node can be understood as the updated tag DOM tree corresponding to the I-th web page tag (referred to as the target DOM tree corresponding to the I-th tag, i.e., corresponding to the second DOM tree mentioned above), and the structural weight of the I-th tag (also the structural weight of the i-th target tree node) is determined according to the tag DOM tree corresponding to the I-th tag.

[0079] First, for each target tree node of the target DOM tree corresponding to the I-th tag, the structural weight of the target data node is calculated according to the text length of the text block corresponding to the target tree node and the depth of the target tree node in the DOM tree, as shown in formula (1):

[0080]

[0081] In formula (1), The number of words in the text block corresponding to any target tree node in the target DOM tree corresponding to the I-th tag, where the number of words in the text represents the length information of the text; is the node depth of any target tree node in the target DOM tree corresponding to the I-th tag, representing the hierarchical information of the target tree node in the target DOM tree. In addition, , Indicates the maximum depth (e.g. ). In addition, set , used to ensure that the structural weight of each target tree node obtained is positive and meaningful, is the initial structural weight of the target DOM tree, It represents the initial structural weight of each target tree node. Represents the structural weight of any target tree node in the target DOM tree corresponding to the I-th tag.

[0082] In some embodiments, after obtaining the structural weight of each target tree node in the target DOM tree corresponding to the ith tag, the structural weights of all target tree nodes in the target DOM tree corresponding to the ith tag may be summed up and the calculation result may be used as the structural weight of the ith tag, i.e., the structural weight of the ith target tree node. The calculation process can refer to formula (2).

[0083]

[0084] In formula (2), Z represents the number of all target tree nodes in the target DOM tree corresponding to the I-th tag.

[0085] In some embodiments, the intuitive meaning implied by the formula for calculating the structural weight is: if a target tree node in the target DOM tree has a greater depth in the target DOM tree, the lower the level of the target tree node, the smaller the impact of the text information corresponding to the target tree node on the entire target DOM tree, and vice versa, the closer a target tree node is to the root node of the target DOM tree, the greater the impact of the text information corresponding to the target tree node on the entire target DOM tree. In addition, if a target tree node in the target DOM tree has more text characters corresponding to it, the greater the weight of the target tree node; conversely, the fewer the characters, the smaller the weight.

[0086] In an optional embodiment, before filtering out the text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, the text extraction method further includes: for the i-th target tree node in any target DOM tree, determining a tag DOM tree with the i-th target tree node as a root node as a third DOM tree, wherein the tag DOM tree includes target tree nodes and non-target tree nodes, and i is an integer greater than or equal to 1; obtaining the number of target tree nodes and the number of non-target tree nodes in the third DOM tree; and determining the text weight of the i-th target tree node according to the number of target tree nodes and the number of non-target tree nodes in the third DOM tree and a preset text weight hyperparameter.

[0087] In some embodiments, for a tag DOM tree corresponding to a certain tag, the text weight of the tag can be calculated by the following formula (3):

[0088]

[0089] In formula (3), is the number of target tree nodes in the tag DOM tree of the I-th tag (corresponding to the third DOM tree with the i-th target tree node as the root node), that is, the number of tags with textual meaning in all tags corresponding to the tag DOM tree of the I-th tag. is the number of non-target tree nodes in the tag DOM tree of the I-th tag (corresponding to the third DOM tree with the i-th target tree node as the root node), that is, the number of tags that do not have textual meaning among all tags corresponding to the tag DOM tree of the I-th tag.

[0090] In some embodiments, in formula (3), is the text weight hyperparameter.

[0091] In some embodiments, it can be seen from formula (3) that the number of tags with textual meaning is positively correlated with the text weight, while the number of meaningless tags is negatively correlated with the text weight. According to the technical solution of the present application, tags that truly have textual meaning can be retained by amplifying the penalty term. It is worth noting that the text weight calculated using formula (3) is allowed to be negative.

[0092] It should be noted that the process of obtaining the tag DOM tree includes: generating an initial DOM tree according to the web page source code, wherein each web page tag in the web page source code corresponds to a tree node in the initial DOM tree; filtering non-key web page tags in the web page source code, and deleting the tree nodes corresponding to the non-key web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that the text block corresponding to the non-key web page tag belongs to the web page body is lower than a preset probability; determining L tag DOM trees according to the L tree nodes in the first DOM tree through a breadth search strategy and a depth search strategy, wherein L is an integer greater than or equal to 1, the root node of each tag DOM tree is a tree node of the first DOM tree, the breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree.

[0093] The specific implementation method of how to obtain the tag DOM tree has been described in the related content of the above-mentioned embodiment, and will not be repeated here.

[0094] In an optional embodiment, based on the text weight and structural weight of each target tree node, the main text tree nodes are screened from all the target tree nodes, including: obtaining preset hyperparameters of the text weight; calculating the product of the text weight of each target tree node and the preset hyperparameter to obtain the target text weight of each target tree node; calculating the product of the target text weight and the structural weight of each target tree node to obtain the target weight of each target tree node; based on the target weight of each target tree node, the main text tree nodes are screened from all the target tree nodes.

[0095] In some embodiments, after obtaining the text weight and structure weight of each target tree node, the target weight of the target tree node can be calculated by the following formula (4): (Also called comprehensive weight).

[0096]

[0097] In formula (4), Preset hyperparameters for text weights of target tree nodes. is the text weight of the target tree node, is the structural weight of the target tree node.

[0098] In an optional embodiment, according to the target weight of each target tree node, the main text tree node is obtained from all the target tree nodes, including: according to the target weight of each target tree node, all the target tree nodes are sorted, and the main text tree nodes are obtained from all the target tree nodes according to the sorting result; or, according to the target weight of each target tree node, the target tree nodes whose target weights are greater than a preset threshold are selected from all the target tree nodes as the main text tree nodes.

[0099] In some embodiments, after obtaining the target weights corresponding to all target tree nodes, all target tree nodes can be sorted from large to small according to the target weights and compared with a preset threshold. When the target weight of a target tree node is greater than the preset threshold, the target tree node is used as the main text tree node.

[0100] In some embodiments, another method for determining the text tree nodes can be provided, that is, all target tree nodes are sorted from large to small according to the target weights, and then a preset number of target tree nodes with larger target weights are selected as the text tree nodes according to the sorting results.

[0101] In some embodiments, another method for determining the text tree node may be provided, which is a combination of the above two methods, as shown in formula (5):

[0102]

[0103] In formula (5), is the preset threshold mentioned above, G is the number of text tree nodes that need to be selected, For the identified text label node, the meaning of formula (5) is: according to the weight of each target tree node , select G target tree nodes with the largest target weights from all target tree nodes as the main text tree nodes, and the target weights of the main text tree nodes must be greater than a preset threshold.

[0104] In an optional application scenario, according to the technical solution of this application, useful information can be extracted from the web page, ignoring non-text content such as advertisements, navigation bars, footers, etc. This method can also be applied to threat intelligence acquisition and other web page information acquisition fields, helping security analysts to quickly locate and extract information about network security threats from a large number of network resources.

[0105] As can be seen from the above content, this application solves the problem that the web page text cannot be correctly identified in a complex network environment. The technical solution of this application comprehensively considers text and structural factors, does not require pre-training of the model, has higher accuracy and faster speed. In addition, the web page text recognition method proposed in this application can be flexibly configured to adapt to different web page frameworks by modifying the filter tag set, weight calculation factor and weight threshold, thereby ensuring the accuracy and applicability of recognition. It can be transplanted to other web page text extraction scenarios and has good versatility.

[0106] In an optional embodiment, Figure 3 is a schematic diagram of an optional text extraction system according to an embodiment of the present application, such as Figure 3 As shown, it includes: a web page collection component, a tag filtering component, a DOM tree node screening component, a weight calculation component, a text node determination component, and a text extraction component.

[0107] In some embodiments, Figure 4 is a flowchart of another optional text extraction method according to an embodiment of the present application, such as Figure 4 As shown, first, the HTML source code of the web page is collected through the web page collection component, and then the tag filtering component is used to filter the tags and clean up the non-key tags; then, the tag DOM tree corresponding to the remaining key tags is determined by the DOM tree node screening component, and the meaningless non-key tags are further filtered through the breadth search strategy and the depth search strategy, and the tree nodes corresponding to the non-key tags in the tag DOM tree are deleted to obtain the target DOM tree. Subsequently, the text weight and structure weight of each target tree node in the target DOM tree are calculated by the weight calculation component, and the comprehensive weight of each target tree node is calculated. Then, all the target tree nodes are sorted from large to small according to the comprehensive weight through the text node determination component, and the target tree nodes with a comprehensive weight greater than a preset threshold are used as the text tree nodes. Finally, the text corresponding to the text tree nodes are extracted from the web page as the web page text through the text extraction component.

[0108] From the above content, it can be seen that a node screening method for a DOM tree is provided, including: identifying a tree node whose corresponding text block is non-empty text and whose web page tag corresponding to at least one layer of upper-level nodes is a target web page tag (a meaningful tag) as a target tree node, and then calculating the text weight and structure weight for each target tree node, thereby comprehensively determining which target tree nodes belong to the main text tree node in terms of the two dimensions of text information and structure information, and finally extracting the text block corresponding to the main text tree node from the web page. Compared with the statistical method of extracting the text content of a web page in the prior art, the present application comprehensively determines the main text information from at least the following four dimensions:

[0109] The first dimension is whether the text block corresponding to the DOM tree node is empty text;

[0110] The second dimension is whether the web page tag corresponding to at least one layer of upper-level nodes (such as parent nodes and grandparent nodes) of the DOM tree node is the target web page tag;

[0111] The third dimension is the text weight of the DOM tree node;

[0112] The fourth dimension is the structural weight of the DOM tree node;

[0113] By providing more reference dimensions, the present application can improve the accuracy of extracting the body content of a web page, thereby solving the technical problem of inaccurate extraction of the body content of a web page caused by extracting the text content of a web page in a statistical manner in the prior art.

[0114] According to another aspect of the present application, an embodiment of the present application further provides a text extraction device, wherein: Figure 5 is a schematic diagram of an optional text extraction device according to an embodiment of the present application, such as Figure 5 As shown, the text extraction device includes: a first processing unit 501, a second processing unit 502, and a third processing unit 503.

[0115] Among them, the first processing unit 501 is used to determine at least one target DOM tree based on the web page tags in the web page source code, wherein each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is a non-empty text, and the web page tag corresponding to at least one layer of the upper node of the target tree node is the target web page tag, and the target web page tag includes a target element for circling the web page text; the second processing unit 502 is used to filter out the text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the web page, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the web page; the third processing unit 503 is used to extract the text block corresponding to the text tree node from the web page.

[0116] In some embodiments, the first processing unit 501 includes: a first processing subunit, a second processing subunit, and a first determining subunit. The first processing subunit is used to generate an initial DOM tree according to the web page source code, wherein each web page tag in the web page source code corresponds to a tree node in the initial DOM tree; the second processing subunit is used to filter non-critical web page tags in the web page source code, and delete the tree nodes corresponding to the non-critical web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that the text block corresponding to the non-critical web page tag belongs to the web page body is lower than a preset probability; the first determining subunit is used to detect the tree nodes belonging to the target tree nodes in the first DOM tree, and determine the target DOM tree according to the detection result.

[0117] In some embodiments, the first determination subunit includes: a first determination module and a first processing module. The first determination module is used to determine L tag DOM trees according to L tree nodes in the first DOM tree through a breadth search strategy and a depth search strategy, wherein L is an integer greater than or equal to 1, the root node of each tag DOM tree is a tree node of the first DOM tree, the breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree; the first processing module is used to detect whether each tree node in each tag DOM tree belongs to a target tree node, and according to the detection result, delete the tree nodes that do not belong to the target tree node in each tag DOM tree to obtain the target DOM tree.

[0118] In some embodiments, the text extraction device further includes: a fourth processing unit, a first acquisition unit, a fifth processing unit, a first determination unit, and a fifth processing unit. The fourth processing unit is used to determine the target DOM tree with the i-th target tree node as the root node as the second DOM tree for the i-th target tree node in any target DOM tree; the first acquisition unit is used to obtain the text length of the text block corresponding to each tree node in the second DOM tree; the fifth processing unit is used to use the hierarchical information of each tree node in the second DOM tree in the second DOM tree as the node depth of the tree node; the first determination unit is used to determine the structural weight of each tree node in the second DOM tree according to the node depth of each tree node in the second DOM tree and the text length of the corresponding text block; the fifth processing unit is used to sum the structural weights of all tree nodes in the second DOM tree and use the summed result as the structural weight of the i-th target tree node.

[0119] In some embodiments, the text extraction device further includes: a sixth processing unit, a seventh processing unit, and a second determination unit. The sixth processing unit is used to determine the tag DOM tree with the i-th target tree node as the root node as the third DOM tree for the i-th target tree node in any target DOM tree, wherein the tag DOM tree includes target tree nodes and non-target tree nodes, and i is an integer greater than or equal to 1; the seventh processing unit is used to obtain the number of target tree nodes and the number of non-target tree nodes in the third DOM tree; the second determination unit is used to determine the text weight of the i-th target tree node according to the number of target tree nodes and the number of non-target tree nodes in the third DOM tree and the preset text weight hyperparameter.

[0120] In some embodiments, the second processing unit 502 includes: a first acquisition subunit, a first calculation subunit, a second calculation subunit, and a third processing subunit. The first acquisition subunit is used to acquire the preset hyperparameter of the text weight; the first calculation subunit is used to calculate the product of the text weight of each target tree node and the preset hyperparameter to obtain the target text weight of each target tree node; the second calculation subunit is used to calculate the product of the target text weight and the structure weight of each target tree node to obtain the target weight of each target tree node; the third processing subunit is used to filter out the text tree node from all the target tree nodes according to the target weight of each target tree node.

[0121] In some embodiments, the third processing sub-unit is also used to sort all target tree nodes according to the target weight of each target tree node, and filter out the main text tree nodes from all target tree nodes according to the sorting result; or, according to the target weight of each target tree node, filter out the target tree nodes whose target weights are greater than a preset threshold from all target tree nodes as the main text tree nodes.

[0122] According to another aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the text extraction method provided in any one of the above-mentioned embodiments.

[0123] According to another aspect of the present application, an electronic device is also provided, wherein the electronic device includes one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors execute the text extraction method provided in any one of the above-mentioned embodiments.

[0124] The above-mentioned embodiments or examples disclosed in the present application are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection disclosed in the present application. In the absence of contradiction, each step in a certain embodiment or example in the present application can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0125] The serial numbers of the above-mentioned embodiments of the present application are for description only and do not represent the advantages or disadvantages of the embodiments.

[0126] In the above embodiments of the present application, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, please refer to the relevant description of other embodiments.

[0127] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0128] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0129] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0130] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc., which can store program code.

[0131] The above is only a preferred implementation of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A text extraction method, characterized in that: include: Determine at least one target DOM tree according to the web page tag in the web page source code, wherein each tree node in the target DOM tree is a target tree node, the text block corresponding to the target tree node is a non-empty text, and the web page tag corresponding to at least one upper node of the target tree node is a target web page tag, and the target web page tag includes a target element for circumscribing the web page body; According to the text weight and structure weight of each target tree node, a text tree node is obtained by screening from all the target tree nodes, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the webpage, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the webpage, wherein the structure weight of the target tree node is related to the depth of the target tree node in the target DOM tree and the number of text words corresponding to the target tree node, and the structure weight of the target tree node is negatively correlated with the depth of the target tree node in the target DOM tree, and positively correlated with the number of text words corresponding to the target tree node; Extracting the text block corresponding to the text tree node from the web page; Wherein, determining at least one target DOM tree according to web page tags in the web page source code includes: filtering non-key web page tags in the web page source code, and deleting tree nodes corresponding to the non-key web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that the text block corresponding to the non-key web page tag belongs to the web page body is lower than a preset probability; detecting tree nodes belonging to target tree nodes in the first DOM tree, and determining the target DOM tree according to the detection result.

2. The text extraction method according to claim 1, characterized in that: Before determining at least one target DOM tree according to the web page tags in the web page source code, the text extraction method further includes: The initial DOM tree is generated according to the web page source code, wherein each web page tag in the web page source code corresponds to a tree node in the initial DOM tree.

3. The text extraction method according to claim 2, characterized in that: Detecting a tree node in the first DOM tree that belongs to a target tree node, and determining the target DOM tree according to the detection result, including: Determine L tag DOM trees according to L tree nodes in the first DOM tree by using a breadth search strategy and a depth search strategy, wherein L is an integer greater than or equal to 1, and a root node of each tag DOM tree is a tree node of the first DOM tree, the breadth search strategy is used to determine the maximum breadth of the tag DOM tree, and the depth search strategy is used to determine the maximum depth of the tag DOM tree; It is detected whether each tree node in each of the tag DOM trees belongs to the target tree node, and according to the detection result, the tree nodes in each of the tag DOM trees that do not belong to the target tree node are deleted to obtain the target DOM tree.

4. The text extraction method according to claim 1, characterized in that: Before filtering out the text tree nodes from all the target tree nodes according to the text weight and the structure weight of each target tree node, the text extraction method further comprises: For the i-th target tree node in any target DOM tree, determine the target DOM tree with the i-th target tree node as the root node as the second DOM tree, where i is an integer greater than or equal to 1; Obtaining the text length of the text block corresponding to each tree node in the second DOM tree; Using the level information of each tree node in the second DOM tree as the node depth of the tree node; determining a structural weight of each tree node in the second DOM tree according to a node depth of each tree node in the second DOM tree and a text length of a corresponding text block; The structural weights of all tree nodes in the second DOM tree are summed up and the sum result is used as the structural weight of the i-th target tree node.

5. The text extraction method according to claim 1, characterized in that: Before filtering out the text tree nodes from all the target tree nodes according to the text weight and the structure weight of each target tree node, the text extraction method further comprises: For the i-th target tree node in any target DOM tree, determine a tag DOM tree with the i-th target tree node as a root node as a third DOM tree, wherein the tag DOM tree includes the target tree node and the non-target tree node, and i is an integer greater than or equal to 1; Obtain the number of target tree nodes and the number of non-target tree nodes in the third DOM tree; The text weight of the i-th target tree node is determined according to the number of target tree nodes and the number of non-target tree nodes in the third DOM tree and a preset text weight hyperparameter.

6. The text extraction method according to any one of claims 1 to 5, characterized in that: According to the text weight and structure weight of each target tree node, the text tree node is screened from all the target tree nodes, including: Obtaining preset hyperparameters of the text weight; Calculate the product of the text weight of each target tree node and the preset hyperparameter to obtain the target text weight of each target tree node; Calculate the product of the target text weight and the structure weight of each target tree node to obtain the target weight of each target tree node; According to the target weight of each target tree node, the text tree node is screened from all the target tree nodes.

7. The text extraction method according to claim 6, characterized in that: According to the target weight of each target tree node, a text tree node is obtained by screening from all the target tree nodes, including: According to the target weight of each target tree node, all the target tree nodes are sorted, and according to the sorting result, the text tree node is obtained by screening from all the target tree nodes; or, According to the target weight of each target tree node, target tree nodes whose target weight is greater than a preset threshold are screened from all the target tree nodes as the main text tree nodes.

8. A text extraction device, characterized in that: include: A first processing unit is used to determine at least one target DOM tree according to a web page tag in a web page source code, wherein each tree node in the target DOM tree is a target tree node, a text block corresponding to the target tree node is a non-empty text, and a web page tag corresponding to at least one upper node of the target tree node is a target web page tag, and the target web page tag includes a target element for circumscribing a web page body; A second processing unit is used to filter out a text tree node from all the target tree nodes according to the text weight and structure weight of each target tree node, wherein the text weight represents the importance of the text block corresponding to the target tree node in the semantic expression of the webpage, and the structure weight represents the importance of the text block corresponding to the target tree node in the text layout structure of the webpage, wherein the structure weight of the target tree node is related to the depth of the target tree node in the target DOM tree and the number of text words corresponding to the target tree node, and the structure weight of the target tree node is negatively correlated with the depth of the target tree node in the target DOM tree, and positively correlated with the number of text words corresponding to the target tree node; A third processing unit, configured to extract a text block corresponding to the text tree node from the web page; Wherein, determining at least one target DOM tree according to web page tags in the web page source code includes: filtering non-key web page tags in the web page source code, and deleting tree nodes corresponding to the non-key web page tags in the initial DOM tree to obtain a first DOM tree; wherein the probability that the text block corresponding to the non-key web page tag belongs to the web page body is lower than a preset probability; detecting tree nodes belonging to target tree nodes in the first DOM tree, and determining the target DOM tree according to the detection result.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located executes the text extraction method according to any one of claims 1 to 7.

10. An electronic device, characterized in that: It comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the text extraction method described in any one of claims 1 to 7.

Images