Web page text area and text information processing method, system, device, and medium

Abstract

The present application provides a web page text area and text information processing method, system, device, and medium, which are applied to the field of information processing technology, wherein the web page text area processing scheme includes: obtaining the DOM tree corresponding to the web page to be processed; based on the visual information and content information of the target node in the DOM tree, searching each node layer by layer from top to bottom along the DOM tree, and jointly determining whether the target node currently searched belongs to the DOM node representing the web page text area; extracting the DOM node representing the web page text area as the candidate node corresponding to the text area. By identifying and extracting the web page text area based on visual information, not only can the text area be accurately and efficiently identified and extracted from the web page, but also there is no need to target a website with a specific web page structure, and it has strong versatility, better robustness, and higher accuracy.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and specifically to a web page text area and text information processing method, system, device, and medium. Background Art

[0002] With the development of Internet technology, the number of Internet sites and page information has increased explosively, and the Internet has become the main way for people to obtain information. However, due to commercial factors, these sites, while providing valuable information to people, also contain other information, such as advertisements or links to other websites. These contents will not only reduce people's reading experience, but also reduce the efficiency of search engines.

[0003] In the process of extracting web page information, the existing solutions can only process specific web pages, are not very universal, and have low extraction efficiency and accuracy.

[0004] Therefore, a new solution for web page information extraction is needed. Summary of the invention

[0005] In view of this, the embodiments of this specification provide a web page text area and text information processing method, system, device, and medium, which can be applicable to multi-record type web pages, such as extracting web page text information from multi-record type web pages, and accurately identifying the area where the text is located from the web page, with better robustness and higher accuracy.

[0006] The embodiments of this specification provide the following technical solutions:

[0007] The embodiment of this specification provides a method for processing a web page text area, including:

[0008] Obtaining a DOM tree corresponding to the web page to be processed, wherein a DOM node of the DOM tree contains visual information and content information corresponding to the DOM node;

[0009] Based on the visual information of the target node in the DOM tree, determining the visual area ratio corresponding to the target node, wherein the visual area ratio includes the ratio of the visual area of ​​the target node to the visual area of ​​the web page to be processed;

[0010] Based on the content information of the target node in the DOM tree, determine the content length ratio and symbol density corresponding to the target node, wherein the content length ratio includes the ratio of the sum of the text lengths of the target node and all its descendant nodes to the sum of the text lengths of all nodes on the web page to be processed, and the symbol density includes the ratio of the text length excluding links in the target node to the number of symbols;

[0011] Searching each node layer by layer from top to bottom along the DOM tree, so as to jointly determine whether the target node currently being searched belongs to the DOM node representing the body area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density;

[0012] When it is determined that the target node currently being searched belongs to a DOM node representing the text area of ​​a web page, the DOM node representing the text area of ​​the web page is extracted as a candidate node corresponding to the text area.

[0013] The embodiment of the present specification also provides a method for processing webpage text information, including:

[0014] Obtaining target candidate nodes corresponding to each text area in the web page to be processed, wherein the target candidate nodes are candidate nodes corresponding to the text areas obtained after the web page to be processed is processed by the web page text area processing method described in any embodiment of this specification;

[0015] Clustering all child nodes under the target candidate node to obtain a first clustering result corresponding to the target candidate node;

[0016] The child node corresponding to the largest cluster in the first clustering result is determined as a target data record, wherein the target data record is a data record containing text information.

[0017] The embodiment of the present specification also provides a web page text area processing system, including:

[0018] A first acquisition module acquires a DOM tree corresponding to the web page to be processed, wherein a DOM node of the DOM tree contains visual information and content information corresponding to the DOM node;

[0019] A first calculation module determines a visual area ratio corresponding to a target node in a DOM tree based on visual information of the target node, wherein the visual area ratio includes a ratio of a visual area of ​​the target node to a visual area of ​​the web page to be processed;

[0020] A second calculation module determines, based on the content information of the target node in the DOM tree, a content length ratio and a symbol density corresponding to the target node, wherein the content length ratio includes a ratio of the sum of the text lengths of the target node and all its descendant nodes to the sum of the text lengths of all nodes on the web page to be processed, and the symbol density includes a ratio of the text length excluding links in the target node to the number of symbols;

[0021] The first traversal module searches each node layer by layer from top to bottom along the DOM tree to jointly determine whether the target node currently being searched belongs to the DOM node representing the body area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density;

[0022] The first candidate module, when determining that the target node currently being searched belongs to a DOM node representing the body area of ​​a web page, extracts the DOM node representing the body area of ​​the web page as a candidate node corresponding to the body area.

[0023] The embodiment of the present specification also provides a web page text information processing system, including:

[0024] A second acquisition module, acquiring target candidate nodes corresponding to each text area in the web page to be processed, wherein the target candidate nodes are candidate nodes corresponding to the text area obtained after the web page to be processed is processed by the web page text area processing method according to any one of claims 1 to 8;

[0025] A clustering module clusters all child nodes under the target candidate node;

[0026] The second candidate module determines the child node corresponding to the largest cluster in the clustering result as the target data record, wherein the target data record is a data record containing text information.

[0027] The embodiment of the present specification also provides an electronic device, including:

[0028] At least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: a method for processing a web page body area, or a method for processing web page body information as described in any one of the embodiments in this specification.

[0029] An embodiment of the present specification also provides a computer storage medium, which stores computer executable instructions. When the computer executable instructions are executed by a processor, the web page text area processing method or the web page text information processing method described in any embodiment of the present specification is executed.

[0030] Compared with the prior art, the at least one technical solution adopted in the embodiments of this specification can achieve the following beneficial effects:

[0031] By using the visual information and content information of the DOM nodes, all DOM nodes that may indicate the text area of ​​the web page are searched layer by layer from top to bottom along the DOM tree, and these nodes are added to the candidate text set as the text area, so as to facilitate the subsequent text information extraction based on these text areas. Therefore, in the process of identifying and extracting the text area corresponding to the web page text information based on visual information, not only can the text area be accurately and efficiently identified and extracted from the web page, but also there is no need to target a website with a specific web page structure, it has strong versatility, better robustness and higher accuracy, and is well suitable for multi-record web page information extraction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0033] Figure 1 It is a structural schematic diagram of a method for processing the body text area of ​​a web page in this application;

[0034] Figure 2 It is a flow chart of a method for processing the body area of ​​a web page in this application;

[0035] Figure 3 It is a structural diagram of a web page text area processing system in this application;

[0036] Figure 4 It is a flow chart of a web page text information processing method in this application;

[0037] Figure 5 It is a schematic diagram of the experimental effect corresponding to the weighted combination of similarity in a webpage text information processing method in the present application;

[0038] Figure 6 It is a structural diagram of a web page text information processing system in this application;

[0039] Figure 7 It is a flow chart of text processing of a web page in this application;

[0040] Figure 8 It is a schematic diagram of the local structure of the DOM tree corresponding to the loaded web page in this application;

[0041] Fig. 9 This is a schematic diagram of using xpath expressions to express path information of nodes in a DOM tree in this application;

[0042] Fig.10 is a schematic diagram of the visual information in this application;

[0043] Fig.11 is a schematic diagram of the text area determination results in this application;

[0044] Fig.12 It is a structural diagram of the data record determination result in this application. DETAILED DESCRIPTION

[0045] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0046] The following describes the implementation methods of the present application through specific examples, and those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. The present application can also be implemented or applied through other different specific implementation methods, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present application. It should be noted that, in the absence of conflict, the following embodiments and the features in the embodiments can be combined with each other. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work belong to the scope of protection of the present application.

[0047] It should be noted that various aspects of the embodiments within the scope of the appended claims are described below. It should be apparent that the aspects described herein can be embodied in a wide variety of forms, and any specific structure and / or function described herein is merely illustrative. Based on the present application, it should be understood by those skilled in the art that an aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number and aspect described herein can be used to implement the device and / or practice the method. In addition, other structures and / or functionalities other than one or more of the aspects described herein can be used to implement this device and / or practice this method.

[0048] It should also be noted that the illustrations provided in the following embodiments are only schematic illustrations of the basic concept of the present application. The drawings only show components related to the present application rather than being drawn according to the number, shape and size of components in actual implementation. In actual implementation, the type, quantity and proportion of each component may be changed arbitrarily, and the component layout may also be more complicated.

[0049] Additionally, in the following description, specific details are provided to facilitate a thorough understanding of the examples. However, it will be understood by those skilled in the art that the examples can be practiced without these specific details.

[0050] At present, web pages can be divided into single-record web pages and multi-record web pages according to their layout. The extraction of information from single-record web pages is relatively easy to achieve, and the existing web page information extraction scheme can already solve the problem of information extraction from single-record web pages. However, the information extraction effect on multi-record pages is not good.

[0051] Among them, the information extraction methods for multi-record web pages are divided into two major technical routes: DOM structure-based extraction solutions and vision-based extraction solutions. DOM structure-based extraction is to parse the HTML document into the corresponding DOM tree, and then create extraction rules based on the syntax of the DOM tree. It has high accuracy and performance, but when the DOM tree of the web page is deep and contains a large number of DOM nodes, the time and space complexity of this type of algorithm is high. The vision-based web page information extraction method has gradually emerged in the current web page information extraction technology and is a more advanced solution route. Among them, the web page information extraction method based on visual information first pre-renders the target web page through the browser interface or kernel, and then extracts web page data records based on the visual rules of the web page. However, due to the difficulty of technical implementation, the existing solutions are more targeted at specific web pages for extraction, with poor versatility, and the accuracy and efficiency of web page information extraction need to be improved.

[0052] In view of this, the inventors conducted in-depth research and improvement exploration on various websites, web page structures and their page information, as well as various extraction schemes, and found that: when faced with various web pages, the existing schemes are unable to accurately and effectively identify and extract the area where the main text information of the web page is located (which can be called the main text area), that is, the main text area cannot be accurately and efficiently identified and extracted, resulting in the existing schemes having problems such as low versatility, poor accuracy, and low efficiency in practical applications.

[0053] Therefore, if the body area where the web page text information is located can be efficiently identified, extracted and processed, the versatility, accuracy, efficiency, etc. of the web page text information extraction solution will inevitably be substantially improved.

[0054] Based on this, the embodiment of this specification proposes a processing scheme for identifying and extracting the text area where the web page text information is located based on visual information: Figure 1As shown, each node on the DOM tree corresponds to an area on the web page, and the text area is usually composed of data records arranged in sequence, has a large visual area on the page, and contains rich text content. At the same time, the web page text generally has corresponding punctuation marks, so these features of the text area can be used together for the identification and extraction of the text area. Specifically, using the visual information and content information of the DOM node, all DOM nodes that may indicate the text area of ​​the web page are searched layer by layer from top to bottom along the DOM tree, and these nodes are added as candidate nodes to the candidate text set representing the DOM node corresponding to the text area. During the search, if there is a child node in the current candidate node that simultaneously meets the three conditions of visual area, content length and symbol density, then the child node can be recorded as a DOM node that can represent the text area of ​​the web page, so that the relevant nodes that can represent the text area in the web page (i.e., the DOM node corresponding to the text area) can be accurately and quickly extracted.

[0055] The technical solutions provided by various embodiments of the present application are described below in conjunction with the accompanying drawings.

[0056] like Figure 2 As shown, the embodiment of this specification provides a method for processing a web page body area, which may include:

[0057] Step S202: Obtain a DOM tree corresponding to the web page to be processed, wherein a DOM node of the DOM tree contains visual information and content information corresponding to the DOM node.

[0058] In practice, according to the W3C standard, the main content of a web page is generally in the body tag, so the body tag can be taken as the root node to construct the DOM tree.

[0059] Content information is the main part of an HTML web page, which may include tags, text, links, media and other data information in a web page document, and can be directly obtained from an HTML document through natural language processing. Tags refer to HTML semantic tags, which usually appear in pairs, such as 、

[0060]

[0061]

[0062]

[0063]

[0064]

[0065]

[0066]

[0067]

[0068]

[0069]

[0070]

[0071]

[0072]

[0073]

[0074]

[0075]

[0076]

[0077]

[0078]

[0079]

[0080]

[0081]

[0082]

[0083]

[0084]

[0085]

[0086]

[0087]

[0088]

[0089]

[0090]

[0091]

[0092]

[0093]

[0094]

[0095]

[0096]

[0097]

[0098] Figure 3

[0099]

[0100]

[0101]

[0102]

[0103]

[0104]

[0105]

[0106]

[0107]

[0108]

[0109]

[0110]

[0111]

[0112]

[0113]

[0114]

[0115]

[0116]

[0117]

[0118]

[0119]

[0120] Figure 4

[0121]

[0122]

[0123]

[0124]

[0125]

[0126]

[0127]

[0128]

[0129]

[0130]

[0131]

[0132]

[0133]

[0134]

[0135]

[0136]

[0137]

[0138] Figure 5

[0139]

[0140]

[0141]

[0142]

[0143]

[0144]

[0145]

[0146]

[0147]

[0148] Figure 6

[0149]

[0150]

[0151]

[0152]

[0153]

[0154]

[0155]

[0156]

[0157]

[0158]

[0159]

[0160]

[0161]

[0162]

[0163]

[0164] Figure 7

[0165]

[0166]

[0167]

[0168]

[0169]

[0170]

[0171] Figure 8 Fig. 9

[0172] Fig. 9

[0173]

[0174]

[0175] Fig.10

[0176]

[0177]

[0178]

[0179]

[0180]

[0181]

[0182]

[0183]

[0184]

[0185]

[0186]

[0187]

[0188]

[0189] Fig.11

[0190]

[0191]

[0192]

[0193]

[0194]

[0195]

[0196]

[0197]

[0198]

[0199]

[0200]

[0201]

[0202]

[0203] Fig.12

[0204]

[0205]

[0206]

[0207]

[0208]

[0209]

[0210]

[0211]

[0212]

[0213]

[0214]

[0215]

[0216]

[0217]

[0218] HTML tags; text refers to the text embedded in the node in the DOM tree (ContentText); link refers to the number of links in the text (LinkNum); media refers to pictures, multimedia, etc. Visual information refers to the visual presentation of a web page, where visual information may include CSS attribute information in terms of text, box model, layout, and background. Text attributes may refer to attributes that characterize text features in visual presentation, and may include indicators such as font color, font type, font size, font weight, and font family; box model may refer to attributes that characterize element features in visual presentation, and may include indicators such as element width, element height, element inner margin, element border, element outer margin, and element size calculation mode; layout may refer to attributes that characterize the relationship features between elements in visual presentation, and may include indicators such as the offset between the boundaries of the parent element (relative offset of the element), the offset between the outer margin boundary of the element and the boundary of the page (absolute offset of the element), the content overflow rule of the element, and the content alignment within the element (horizontal and vertical); background may refer to attributes that characterize background features in visual presentation, and may include indicators such as background color, background image, the starting position of the background image, the repetition rule of the background image, and the drawing size of the background image. It should be noted that each web page can be parsed into a DOM tree, all tags are nodes, and text and pictures are leaf nodes. Here, when obtaining the DOM tree, it can be the DOM tree data information input from the outside, or it can be the DOM tree corresponding to the web page obtained by processing the web page locally. Step S204, based on the visual information of the target node in the DOM tree, determine the visual area ratio corresponding to the target node, wherein the visual area ratio includes the ratio of the visual area of ​​the target node to the visual area of ​​the web page to be processed. In implementation, each node on the DOM tree corresponds to an area on the web page, so the DOM node is represented on the page as an area with a certain size. Usually the body area has a large visual area, and the area size can be used to determine whether the node may belong to the node indicating the body area, wherein the visual area Area_node of the node on the page can be obtained by multiplying the width attribute and the height attribute of the node. The page area Area_page of the web page can be obtained by multiplying the width attribute and the height attribute of the body node. Therefore, the calculation formula of the visual area ratio Area_rate of the node is: Area_rate=Area_node / Area_page. The visual area ratio corresponding to the node can be accurately obtained through visual information. After obtaining the visual area ratio, the visual area ratio can be used to determine that the node may belong to the node corresponding to the main text area, and then the node that may belong to the main text area can be selected from many nodes.Step S206, based on the content information of the target node in the DOM tree, determine the content length ratio and symbol density corresponding to the target node, wherein the content length ratio includes the ratio of the sum of the text lengths of the target node and all its descendant nodes to the sum of the text lengths of all nodes on the to-be-processed web page, and the symbol density includes the ratio of the text length of the target node excluding links to the number of symbols. In implementation, the body text area of ​​a web page usually contains the most abundant information on the web page, and generally has a longer content length, while the content length of the non-body text area is usually shorter, so the content length can be used to reflect the possibility that the node belongs to the body text area. Specifically, the content length ratio is used to measure the content length. Among them, the content length ratio Cont_rate of a node can be calculated by the formula: Cont_rate=Cont_node / Cont_page. Cont_node is the sum of the text lengths of the node and all its descendant nodes, and Cont_page is the total text length of all nodes on the page. In practice, non-text usually has fewer words, and there may be no symbols in non-text, such as some navigation information belonging to non-text, while the text usually contains more words and corresponding symbols, such as news content, discussion content, etc. Therefore, the symbol density can be used to determine whether a node belongs to the corresponding node of the text area, where the symbol density is the ratio of the number of words to the number of symbols in the node. Here, the symbol density of the node is defined as the ratio of the length of the text without links to the number of symbols, where LinkCont_node is recorded as the sum of the length of the text with links in the node and all its descendant nodes, and sym_node represents the number of punctuation marks in the text of the node and all its descendant nodes. Then the symbol density sd_rate of the node is: sd_rate=(Cont_node-LinkCont_node) / (sym_node+1). By extracting two types of parameters (i.e., content length ratio and symbol density) that can characterize the possible indication of the text area of ​​the node from the content information, the possibility of the node belonging to the corresponding node of the text area can be quickly and accurately determined based on these two types of parameters. Step S208, searching each node layer by layer from top to bottom along the DOM tree, so as to jointly determine whether the target node currently being searched belongs to the DOM node representing the text area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density. In implementation, when the visual area ratio, content length ratio and symbol density of the node can simultaneously reflect that the node may belong to the text area node (i.e., the node corresponding to the text area, which is not distinguished in this specification), the node can be identified and extracted as a text area node.By using the three parameters of the node's visual area ratio, content length ratio and symbol density to jointly determine whether the current node belongs to the DOM node representing the body area of ​​the web page, not only is the extraction scheme of the corresponding nodes in the body area simplified, the versatility of the extraction scheme when facing different types of web pages is improved, but also the efficiency of identifying and extracting the nodes in the body area is improved, and it can be ensured that the extracted nodes belong to the body area nodes with high accuracy. Step S210, when it is determined that the target node currently being searched belongs to the DOM node representing the body area of ​​the web page, the DOM node representing the body area of ​​the web page is extracted as the candidate node corresponding to the body area. In implementation, the candidate nodes identified and extracted can be formed into a set, which is convenient for traversing the set. Specifically, a candidate set N of the body area can be defined first, and then the root node of the DOM tree is added to the candidate set N, and then all nodes in the candidate set N are traversed in sequence, and when a candidate node is determined during the traversal, the new candidate node is added to the candidate set N. Specifically, for the candidate node currently traversed, based on the three parameters of the visual area ratio, content length ratio and symbol density, it is determined in turn whether each of its child nodes may belong to the candidate node. If it belongs to the candidate node, the child node is added to the candidate set N. In the above steps S202 to S210, based on the visual information and content information of the DOM tree, that is, based on the three parameters of the visual area ratio, content length ratio and symbol density, each node of the DOM tree can be quickly and accurately traversed and identified and extracted, so that the nodes that may belong to the text area are extracted as candidate nodes. In some embodiments, in obtaining the DOM tree corresponding to the web page, the web page can be directly parsed and rendered by the browser or the browser kernel to obtain the DOM tree with visual information and content information. In implementation, the browser can parse the HTML code of the web page according to the grammatical structure, and after parsing and rendering the resources of the web page, provide the script running environment and the visual presentation of the web page, so as to obtain the DOM tree, wherein the DOM nodes in the DOM tree can carry visual information and content information. Specifically, the web page documents and resources corresponding to the web page to be processed are parsed and rendered through the browser or the browser kernel; after the web page to be processed is parsed and rendered, the DOM structure information, content information and visual information corresponding to the web page to be processed are obtained; and the DOM tree corresponding to the web page to be processed is constructed according to the DOM structure information. Directly obtaining the DOM tree with visual information and content information through the browser or the browser kernel can facilitate the processing scheme to construct the DOM tree for the actual web page, and better adapt to the application of different web page types. In some embodiments, the web page resources corresponding to the specified URL can be downloaded directly based on the browser or the browser kernel, so as to facilitate the rapid generation of the DOM tree on the local side.In implementation, before parsing and rendering the webpage document and resource corresponding to the webpage to be processed, the webpage text area processing method further includes: receiving the URL address corresponding to the webpage to be processed through a browser or a browser kernel; and obtaining the webpage document and resource corresponding to the target link according to the target link corresponding to the webpage to be processed. In some implementations, the DOM tree of the webpage can be pre-denoised according to the specific meanings of various tags in the webpage. In implementation, after obtaining the DOM tree corresponding to the webpage to be processed, the webpage text area processing method further includes: deleting the DOM nodes and all their descendant nodes that are not related to the webpage text information from the DOM tree. For example, according to the specific meaning of the HTML tag, the DOM nodes and all their descendant nodes that are not related to the webpage text information are deleted from the DOM tree, such as footer, nav, aside, script, style, svg, meta, etc. In some implementations, when the three parameters of visual area proportion, content length proportion and symbol density are used to jointly determine whether a node belongs to the text area node, the corresponding preset conditions can be set for the three parameters respectively, so that the preset conditions can be used to quickly determine whether the node belongs to the text area node in the node identification and extraction. Specifically, according to the visual area ratio, the content length ratio and the symbol density, jointly determine whether the target node currently being searched belongs to the DOM node representing the text area of ​​the web page, including: determining whether the visual area ratio, the content length ratio and the symbol density corresponding to the target node meet the respective corresponding preset conditions; and if the visual area ratio, the content length ratio and the symbol density simultaneously meet the respective corresponding preset conditions, then determine that the target node currently being searched belongs to the DOM node representing the text area of ​​the web page. In one example, corresponding thresholds can be set for these parameters, and when the actual value is greater than the threshold, it can be determined that the possibility that the current node belongs to the text area node is high. For example, whether the visual area ratio of the current node exceeds the first threshold Ta, if so, the possibility that the current node belongs to the text node is adapted to be judged as high; whether the content length ratio of the current node exceeds the second threshold Tc, if so, the possibility that the current node belongs to the text node is adapted to be judged as high; whether the symbol density exceeds the third threshold Ts, if so, the possibility that the current node belongs to the text node is adapted to be judged as high. Accordingly, only when the relationship between the three parameters and the preset thresholds can simultaneously indicate that the possibility that the current node belongs to the main text node is relatively high, the current node can be extracted and marked as a main text area node, otherwise it will not be identified and extracted as a main text area node. By simultaneously judging the three parameters and the preset conditions, not only is the extraction accuracy high, but also when one of them does not meet the preset conditions, the identification and extraction processing of the node can be quickly ended, and the processing efficiency is guaranteed.In some embodiments, when traversing the DOM tree, an xpath expression can be used to uniquely represent the path information of each node, so as to facilitate the traversal operation of the DOM tree and the positioning and extraction of the text information in the subsequent text area nodes. Specifically, each node is searched layer by layer from top to bottom along the DOM tree, including: each node in the DOM tree is expressed using an xpath expression, wherein the xpath expression is expressed from top to bottom and from left to right in the DOM tree. It should be noted that the xpath expression can also be expressed from top to bottom and from right to left in the DOM tree, which is only exemplified here. In some embodiments, based on the path information of the node represented by the xpath expression, the text area node obtained by identification and extraction can also be represented by an xpath expression, so as to facilitate the subsequent processing of the data of the text area node. In implementation, the DOM node representing the text area of ​​the web page is extracted as the candidate node corresponding to the text area, including: the xpath expression corresponding to the DOM node representing the text area of ​​the web page is extracted as the xpath expression corresponding to the candidate node corresponding to the text area. In some embodiments, in traversing the DOM tree to identify and extract the text area nodes, since the search is performed layer by layer from top to bottom along the root node of the DOM tree, there is usually a nested relationship between the candidate nodes, and the later the DOM node in the set of candidate nodes is, the greater the possibility that it is the real text area. Therefore, by traversing each node in the set of candidate nodes, the most likely text area node can be identified and extracted. Specifically, the web page text area processing method also includes: determining whether the child nodes of each candidate node have been traversed. Among them, if there are nodes that have not been traversed in the candidate set, then for the untraversed nodes, it is simultaneously determined whether the untraversed nodes belong to the text area nodes based on the above three types of parameters. In addition, after traversing all nodes, the traversal process can be terminated, and the identification and extraction of the text area nodes is completed. Based on the same inventive concept, the embodiments of this specification provide a web page text area processing system, electronic device and computer storage medium corresponding to the web page text area processing method described in any of the above embodiments. As shown, a web page text area processing system 300 provided in an embodiment of the present specification may include a first acquisition module 301 , a first calculation module 303 , a second calculation module 305 , a first traversal module 307 , and a first candidate module 309 .Among them, the first acquisition module 301 is used to obtain the DOM tree corresponding to the web page to be processed, wherein the DOM node of the DOM tree contains the visual information and content information corresponding to the DOM node; the first calculation module 303 is used to determine the visual area ratio corresponding to the target node based on the visual information of the target node in the DOM tree, wherein the visual area ratio includes the ratio of the visual area of ​​the target node to the visual area of ​​the web page to be processed; the second calculation module 305 is used to determine the content length ratio and symbol density corresponding to the target node based on the content information of the target node in the DOM tree, wherein the content length ratio includes the target node and all its descendant nodes The ratio of the sum of the text lengths of the target nodes to the sum of the text lengths of all nodes on the web page to be processed, and the symbol density includes the ratio of the text length of the target node excluding the link to the number of symbols; the first traversal module 307 is used to search each node layer by layer from top to bottom along the DOM tree to jointly determine whether the target node currently being searched belongs to the DOM node representing the body area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density; the first candidate module 309 is used to extract the DOM node representing the body area of ​​the web page as the candidate node corresponding to the body area when it is determined that the target node currently being searched belongs to the DOM node representing the body area of ​​the web page. It should be noted that the first calculation module 303 and the second calculation module 305 can be unified into a processing unit for calculating the three parameters, and then the calculation processing unit can obtain the DOM tree and uniformly calculate the three types of parameters. Optionally, obtaining the DOM tree corresponding to the web page to be processed includes: parsing and rendering the web page documents and resources corresponding to the web page to be processed through a browser or a browser kernel; obtaining the DOM structure information, content information and visual information corresponding to the web page to be processed after the web page to be processed is parsed and rendered; and constructing the DOM tree corresponding to the web page to be processed according to the DOM structure information. Optionally, the web page body area processing system may also include: an input unit (not shown in the figure), which is used to perform the following processing operations on the local end: before parsing and rendering the web page documents and resources corresponding to the web page to be processed, receiving the URL address corresponding to the web page to be processed through a browser or a browser kernel; and, according to the target link corresponding to the web page to be processed, obtaining the web page documents and resources corresponding to the target link. Optionally, the web page body area processing system may also include: a first denoising module (not shown in the figure), which is used to perform the following processing operations on the local end: after obtaining the DOM tree corresponding to the web page to be processed, deleting the DOM nodes and all their descendant nodes that are not related to the web page body information from the DOM tree.Optionally, according to the visual area ratio, the content length ratio and the symbol density, jointly determining whether the target node currently being searched belongs to the DOM node representing the text area of ​​the web page, including: determining whether the visual area ratio, the content length ratio and the symbol density corresponding to the target node meet the respective corresponding preset conditions; and if the visual area ratio, the content length ratio and the symbol density simultaneously meet the respective corresponding preset conditions, then determining that the target node currently being searched belongs to the DOM node representing the text area of ​​the web page. Optionally, searching each node layer by layer from top to bottom along the DOM tree, including: expressing each node in the DOM tree using an xpath expression, wherein the xpath expression is expressed in the direction from top to bottom and from left to right in the DOM tree. Optionally, extracting the DOM node representing the text area of ​​the web page as a candidate node corresponding to the text area, including: extracting the xpath expression corresponding to the DOM node representing the text area of ​​the web page as the xpath expression corresponding to the candidate node corresponding to the text area. Optionally, the first traversal module is also used to: determine whether the child nodes of each candidate node have been traversed. It should be noted that other details about the web page text area processing system 300 can refer to the above-mentioned description of the web page text information extraction method, which will not be repeated here. It should be noted that although several units / modules or sub-units / modules of the web page text information extraction system are mentioned in the above detailed description, this division is only exemplary and not mandatory. In fact, according to the implementation of the present application, the features and functions of two or more units / modules described above can be concretized in one unit / module. Conversely, the features and functions of one unit / module described above can be further divided into multiple units / modules to be concretized. Based on the same inventive concept, an embodiment of this specification provides an electronic device for web page text area processing corresponding to the web page text area processing method described in any of the above embodiments, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: a web page text area processing method as described in any embodiment of this specification. Based on the same inventive concept, an embodiment of this specification provides a computer storage medium for extracting web page text information, wherein the computer storage medium stores computer executable instructions, and when the computer executable instructions are executed by a processor, the web page text area processing method provided in any embodiment of this specification is executed. It should be noted that the computer storage medium may include, but is not limited to: a portable disk, a hard disk, a random access memory, a read-only memory, an erasable programmable read-only memory, an optical storage device, a magnetic storage device, or any suitable combination thereof.In a possible implementation, the present application may also provide a form of implementing data processing as a program product, which includes a program code, and when the program product is run on a terminal device, the program code is used to cause the terminal device to execute several steps in the method described in any of the aforementioned embodiments. Among them, the program code for executing the present application can be written in any combination of one or more programming languages, and the program code can be executed completely on the user device, partially on the user device, as an independent software package, partially on the user device, partially on the remote device, or completely on the remote device. Based on the same inventive concept, the embodiments of this specification provide a web page text information processing method, system, electronic device and storage medium to perform application processing of text information based on the text area given in any one of the embodiments of this specification. As shown, the embodiments of this specification provide a web page text information processing method, including: step S402, obtaining target candidate nodes corresponding to each text area in the web page to be processed, wherein the target candidate nodes are candidate nodes corresponding to the text area obtained after the web page to be processed is processed by the web page text area processing method described in any of the aforementioned embodiments of this specification. In implementation, the web page text area processing as described in any of the above embodiments in this specification can be adopted to perform text area identification and extraction processing on the web page to be processed to obtain candidate nodes that may belong to the text area in the DOM tree. It should be noted that the candidate node corresponding to the text area can be the result obtained in advance by the web page text area processing method, or it can be the current result obtained by processing the current web page on the local side, which is not limited here. Therefore, the acquisition method and source of the candidate node are also not limited. Step S404, cluster all child nodes under the target candidate node to obtain the first clustering result corresponding to the target candidate node. In a multi-record complex web page, data records are usually sub-blocks of the text area, and these sub-blocks generally have similar characteristics, such as DOM structure and visual information. Therefore, for each DOM node in the text candidate set, it can be judged in turn whether the area it contains contains multiple sub-blocks with similar DOM structure and visual information. If found, these sub-blocks are considered to be data records. Based on this, all child nodes under the target candidate node can be clustered, and the clustering result can be used to quickly determine whether the target candidate node contains text information (also called data record, which is not distinguished here). Step S406: Determine the child node corresponding to the largest cluster in the first clustering result as the target data record, where the target data record is a data record containing text information. In implementation, the child node corresponding to the largest cluster can be determined as the target data record to ensure the recognition efficiency and accuracy of the text information.In some embodiments, if it is found in the clustering result that each child node of the target candidate node is a separate cluster, it is determined that the content in the area corresponding to the target candidate node does not belong to the target data record. Through the above steps S402 to S406, clustering is performed for the text area nodes, and then the child nodes corresponding to the largest cluster are determined as the target data record, which can adapt to the identification and extraction of text information of different types of web pages, and also improve the identification and extraction efficiency and accuracy of text information. In some embodiments, in clustering, the similarity characteristics between each child node under the current node can be used for clustering. In implementation, before clustering all child nodes under the target candidate node, the similarity between any two child nodes of the target candidate node can be determined first, and then when clustering all child nodes under the target candidate node, efficient and accurate clustering can be performed according to the similarity. In some embodiments, by calculating the similarity between any two child nodes of the node, wherein the similarity can include DOM structure similarity and visual information similarity, and then the similarity between the two child nodes is weighted to obtain the weighted similarity result. Specifically, determining the similarity between any two child nodes of the target candidate node includes: determining the DOM structure similarity and visual information similarity between any two child nodes of the target candidate node, and then determining the similarity result between any two child nodes of the target candidate node based on the weighted calculation of the DOM structure similarity and the visual information similarity, wherein the sum of the weight of the DOM structure similarity and the weight of the visual information similarity is 1. In implementation, the DOM structure similarity can be defined as follows: Assuming that any two child nodes A and B of the node have their corresponding DOM subtree node path sets Path_A and Path_B respectively, the DOM structure similarity dom_sim_AB between the two subtrees can be defined using the Jaccard coefficient, wherein the calculation of the DOM structure similarity dom_sim_AB is:. The definition of visual similarity can be as follows: Assuming any two child nodes A and B of the node, the number of CSS attributes with equal corresponding values ​​in the visual information corresponding to the two child nodes is recorded as CSS_equ, and the total number of all CSS attributes is recorded as CSS_all. The visual information similarity of the two child nodes is recorded as css_sim_AB, where the calculation formula of the visual information similarity css_sim_AB is: css_sim_AB=CSS_equ / CSS_all. The weighted similarity sim_AB between any two child nodes A and B of the node is: sim_AB=a*dom_sim_AB+b*css_sim_AB, where a is the weight of DOM structure similarity, b is the weight of visual similarity, and a+b=1. It should be noted that the weights a and b can be set and adjusted according to actual applications.As shown, when the web page text information processing method provided in the embodiment of this specification is applied to process the web page text information, the experimental results of the weight a and b value combinations are compared, where there are small differences in precision, recall and F1 under different a and b value combinations, and the weight value combinations are not limited here. In some embodiments, after determining the DOM node corresponding to the data record, the DOM subtree can be simplified by streamlining the DOM subtree, which is beneficial to improving the processing accuracy and efficiency of the data record. Specifically, after determining the target data record, the web page text information processing method also includes: extracting the DOM subtree corresponding to each target data record, and then streamlining the DOM subtree according to the visual information of each node in the DOM subtree. In implementation, at least one of the following streamlining operations may be included: ① deleting invisible branches such as (display attribute is none) from the tree structure; ② deleting branches that do not contain text content; ③ searching upward from the leaf nodes of the DOM tree, if all child element nodes of a block-level element node are row-level elements (display is inline and margin-left and margin-right are 0px), then merging its child element nodes into the current node. In some implementations, in HTML, elements are mainly divided into inline elements and block-level elements. Inline elements will not automatically wrap after writing, while block-level elements will automatically wrap after writing. In order to ensure that the content extracted from each data record is more in line with visual rules, it is necessary to first merge the row-level elements into their parent nodes or ancestor nodes, and then extract the data record content. In implementation, streamlining the DOM subtree may include: searching from the leaf nodes of the DOM subtree from bottom to top, and when it is determined that all child element nodes of a block-level element node are row-level elements, merging the child element nodes of the block-level element node into the current block-level element node. In some embodiments, after determining the text area where the target data record is located, the text information can be extracted from the text area. Specifically, the text information in the text area can be obtained by extracting the text content corresponding to the leaf node on the DOM subtree. In some embodiments, when facing different types of web pages, some common noise information can be set for the web page, and then the noise contained in the text information can be removed according to the preset noise information, which is conducive to improving the accuracy of extracting the text information. Specifically, the noise information in the text content can be deleted according to the preset noise information. It should be noted that the noise information is preset, adjusted, etc. for more practical applications. For example, for forum web pages, the noise information that can be set includes "like", "share", "reply", "forward" and other contents, and then by judging whether these contents are contained in the text information, it is extracted and deleted.As shown, the embodiment of this specification also provides a web page text information processing system 500, including: a second acquisition module 501, a clustering module 503 and a second candidate module 505. Among them, the second acquisition module 501 is used to obtain the target candidate node corresponding to each text area in the web page to be processed, wherein the target candidate node is the candidate node corresponding to the text area obtained after the web page to be processed is processed by the web page text area processing method described in any one of claims 1-8; the clustering module 503 is used to cluster all the child nodes under the target candidate node; the second candidate module 505 is used to determine the child node corresponding to the largest cluster in the clustering result as the target data record, wherein the target data record is a data record containing text information. Optionally, the web page text information processing system also includes: a similarity processing module (not shown in the figure), which is used to determine the similarity between any two child nodes of the target candidate node before clustering all the child nodes under the target candidate node, so the clustering module 503 is specifically used to: cluster all the child nodes under the target candidate node according to the similarity. Optionally, determining the similarity between any two child nodes of the target candidate node includes: determining the DOM structure similarity and visual information similarity between any two child nodes of the target candidate node; and determining the similarity result between any two child nodes of the target candidate node based on weighted calculation of the DOM structure similarity and the visual information similarity, wherein the sum of the weight of the DOM structure similarity and the weight of the visual information similarity is 1. Optionally, the web page text information processing system also includes: a pruning module (not shown in the figure), which is used to extract the DOM subtree corresponding to each target data record, and then simplify the DOM subtree according to the visual information of each node in the DOM subtree. Optionally, simplifying the DOM subtree includes: searching from the leaf nodes of the DOM subtree from bottom to top, and when it is determined that all child element nodes of a block-level element node are row-level elements, merging the child element nodes of the block-level element node into the current block-level element node. Optionally, the web page text information processing system further includes: an extraction module (not shown in the figure), which is used to extract the text content corresponding to the leaf nodes on the DOM subtree. Optionally, the web page text information processing system further includes: a second denoising module (not shown in the figure), which is used to delete the noise information in the text content according to preset noise information. Optionally, the second candidate module 505 is also used to: if it is found in the clustering result that each child node of the target candidate node is clustered separately, it is determined that the content in the area corresponding to the target candidate node does not belong to the target data record.The embodiment of this specification also provides an electronic device for web page text information processing, the electronic device includes at least one processor; and a memory connected to the at least one processor in communication; wherein the memory stores instructions executable by the at least one processor, the instructions are executed by the at least one processor, so that the at least one processor can execute: as described in any embodiment of this specification. It should be noted that the description of the electronic device can refer to the description of the aforementioned embodiment, and will not be described here. The embodiment of this specification provides a computer storage medium for web page text information processing, the computer storage medium stores computer executable instructions, and the computer executable instructions are executed by the processor when executed: as described in any web page text information processing method provided in the embodiment of this specification. It should be noted that the description of the computer storage medium can refer to the description of the aforementioned embodiment, and will not be described here. Each embodiment in this specification can be combined in different ways to form different application schemes, and the specific combination can be determined according to the actual application needs, which is not limited here. The following is a schematic description of the combined application scheme with an implementation example, but it should be understood by those skilled in the art that the combined application scheme can still include a variety of different combination schemes. As shown, the embodiment of this specification provides a web page data processing application scheme, and the main processing flow includes the following stages: web page parsing and rendering, obtaining web page information, web page denoising, determining data records, and extracting data record content. For ease of understanding and schematic illustration, the application processing of a web page is used as an example for explanation. The first step is web page parsing and rendering. Here, the URL address is first received, and then the web page corresponding to the URL is parsed and rendered. Among them, take a web page for a topic discussion in a post bar as an example; through a browser or a browser kernel, download the web page document and related resources of the target link, parse and render the web page, and provide a script running environment and a visual presentation of the web page. The second step is to obtain web page information. After parsing and rendering, the web page can obtain the following information, including DOM structure information, text information (i.e., content information) and visual information. The DOM structure information can construct a DOM tree. According to the W3C standard, the body content of a web page is generally in the body tag, so the body tag is taken as the root node to construct the DOM tree. Among them, the structure of the DOM tree can be as shown: the DOM tree can be obtained by parsing the HTML source code of each web page. It should be noted that those skilled in the art can understand that, since the HTML source code of the Tieba webpage is long, the corresponding DOM tree structure is too large, so only a brief diagram is shown. In the subsequent steps, in order to uniquely identify each node in the DOM tree, an xpath expression is used.After the xpath expressions corresponding to the nodes from top to bottom and from left to right in the above DOM tree are expressed in sequence, the results are as shown. In addition, according to the specific meaning of HTML tags, DOM nodes and all their descendant nodes that are not related to the body information of the web page can be deleted from the DOM tree, such as footer, nav, aside, script, style, svg, meta, etc. Content information mainly includes data such as tags, text, links, etc. in web documents. Content information can be obtained from the DOM tree structure, or extracted from HTML using natural speech processing. As shown, visual information mainly includes CSS attribute information in terms of text, box model, layout and background. It should be noted that different DOM trees may have different visual information. Here, the body node is used as an example for a brief schematic description. Therefore, DOM tree structure information, content information (i.e., text information) and CSS information (i.e., visual information) are obtained based on the parsed and rendered web page. The third step is web page denoising. In order to narrow the extraction range of web page information and reduce the interference of noise information, the web page body area can be extracted first. Usually, the text area is composed of data records arranged in sequence, with a large visual area on the page, containing rich text content, and the web page text generally has punctuation marks. Each node on the DOM tree corresponds to an area on the web page. By using the visual information and content information of the DOM node, and searching for all DOM nodes that may indicate the web page text area layer by layer from top to bottom along the DOM tree, these nodes can be added to the candidate text set. The steps for determining the candidate node are as follows: Step (1), define the text area candidate set N, and add the root node of the DOM tree to the candidate set N; Step (2), traverse all nodes of N in the candidate set in turn, and perform the following operations: Sub-step 1), for the current candidate node, determine in turn whether each of its child nodes meets the following three conditions at the same time: Condition ①: Whether the visual area ratio of the child node on the page exceeds the threshold Ta. The area Area_node of the child node can be obtained by multiplying the width attribute and the height attribute of the node. The page area Area_page of the web page can be obtained by multiplying the width attribute and the height attribute of the body node, where the calculation formula for the visual area ratio Area_rate of the child node is: Area_rate=Area_node / Area_page; Condition ②: Whether the content length ratio of the child node exceeds the threshold Tc. Cont_node is the sum of the text length of the child node and all its descendant nodes, and Cont_page is the total text length of all nodes on the page. Among them, the calculation formula for the content length ratio Cont_rate of the child node is: Cont_rate=Cont_node / Cont_page; Condition ③: Whether the symbol density of the child node exceeds the threshold Ts.The symbol density of a child node is the ratio of the length of the text without links to the number of symbols, LinkCont_node is the sum of the length of the text with links in the child node and all its descendant nodes, sym_node represents the number of punctuation marks in the text of the child node and all its descendant nodes, and the calculation formula corresponding to the text density sd_rate of the child node is: sd_rate=(Cont_node-LinkCont_node) / (sym_node+1). Sub-step 2), if there is a child node in the current candidate node that satisfies the above three conditions at the same time, the child node is added to the candidate set N. Step (3), if there is a node in the candidate set that has not been traversed, execute step (2), otherwise exit. Since the candidate set of the main text area is searched layer by layer from top to bottom along the root node of the DOM tree when searching for the main text area, there is a nested relationship between the candidate nodes, and the later the DOM node in the candidate set is, the greater the possibility that it is the real main text area. Therefore, the output corresponding to the main text area determined after the web page denoising process is the main text candidate area set, and the result is shown as follows. The fourth step is to determine the data record. In a multi-record complex web page, data records are usually sub-blocks of the text area, and these sub-blocks generally have similar DOM structures and visual information. In order to find data records, for each DOM node in the text candidate set, it is determined in turn whether the area it contains contains multiple sub-blocks with similar DOM structures and visual information. If found, these sub-blocks are considered to be data records (each sub-block is represented by its child nodes). In determining data records based on the candidate set of text area nodes (also called the text candidate set), the steps are briefly as follows: Step (1): traverse the nodes in the text area candidate set N from back to front; Step (2): For each node in the text candidate set, count the path information of all nodes on its various DOM subtrees in turn. Use Xpath expressions to uniquely represent the path information of each node. Step (3): Calculate the similarity between any two child nodes of the node, including DOM structure similarity and visual information similarity. And convert the similarity between two nodes into distance; the definition and calculation of similarity are briefly as follows: DOM structure similarity: Assume that for any two child nodes A and B of the node, their corresponding DOM subtree node path sets are Path_A and Path_B respectively, and use the Jaccard coefficient to define the DOM structure similarity between the two subtrees.The formula is as follows: dom_sim_AB=|Path_A∩Path_B| / |Path_A∪Path_B| ; Visual similarity: Assuming that any two child nodes A and B of the node have the same corresponding values, the number of CSS attributes with the same corresponding values ​​in the visual information corresponding to the two nodes is CSS_equ, and the total number of all CSS attributes is CSS_all. Then the visual information similarity css_sim_AB of the two child nodes is: css_sim_AB=CSS_equ / CSS_all; Therefore, the similarity (weighted similarity) between any two child nodes A and B of the node is: sim_AB=a*dom_sim_AB+b*css_sim_AB, where a is the weighting coefficient (i.e., weight) of the DOM structure similarity, and b is the weighting coefficient of the visual similarity. Both a and b are preset constants, and a+b=1; Further, the similarity between any two child nodes A and B of the node can be expressed as the distance dist_AB, where the calculation formula of the distance dist_AB is: dist_AB=1-sim_AB; Step (4): Use the DBSCAN algorithm (density clustering algorithm) to cluster all child nodes under the node. If it is found in the clustering results that each child node is a separate cluster, it is considered that the content in this area is not a data record. Otherwise, the child node corresponding to the largest cluster is regarded as a data record. Step (5): If a data record is found, the root node of the DOM subtree of the data record is added to the data record set and the search ends. Otherwise, continue to traverse the next node and return to step (2) to continue to determine the data record operation. As shown, the output is: the DOM node set corresponding to the data record block. It should be noted that the DOM node set here is only the local result corresponding to the aforementioned URL. The fifth step is to extract the data record content. In HTML, elements are mainly divided into inline elements and block-level elements. Inline elements will not automatically wrap after writing, while block-level elements will automatically wrap after writing. In order to ensure that the content extracted from each data record is more in line with visual rules, the inline elements can be merged into their parent nodes or ancestor nodes before the data record content is extracted. The steps of data record extraction are briefly as follows: Step (1): Extract the DOM subtree of each data record and simplify it according to the visual information of each node in the DOM subtree: ① Delete invisible branches (display attribute is none) from the tree structure; ② Delete branches without text content; ③ Start searching from the leaf nodes of the DOM tree upwards. If all child element nodes of a block-level element node are line-level elements (display is inline and margin-left and margin-right are 0px), merge its child element nodes into the current node; Step (2): Extract the text content of the leaf nodes on the DOM subtree; Step (3): Delete noise information.Determine whether the extracted text content contains "like", "share", "reply", "forward" and other content, if it does, delete it. Here we only take the result of one of the data records as an example for explanation: the data record is: / / body / div[3] / div / div[2] / div / div[4] / div[1] / div[3] / div[2]; the content results extracted from it are Item1 to Item7. It should be noted that the above example is only a combined application scheme formed by combining multiple embodiments in this specification. The combination can also be in other forms, which are not explained one by one here. The various embodiments in this specification are described in a progressive manner, and the same and similar parts between the various embodiments can be referred to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the product embodiment described later, since it corresponds to the method, the description is relatively simple, and the relevant parts can be referred to the partial description of the system embodiment. The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed in the present application should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A method for processing a web page text area, characterized in that: include: Obtaining a DOM tree corresponding to the web page to be processed, wherein a DOM node of the DOM tree contains visual information and content information corresponding to the DOM node; Based on the visual information of the target node in the DOM tree, determining the visual area ratio corresponding to the target node, wherein the visual area ratio includes the ratio of the visual area of ​​the target node to the visual area of ​​the web page to be processed; Based on the content information of the target node in the DOM tree, determine the content length ratio and symbol density corresponding to the target node, wherein the content length ratio includes the ratio of the sum of the text lengths of the target node and all its descendant nodes to the sum of the text lengths of all nodes on the web page to be processed, and the symbol density includes the ratio of the text length excluding links in the target node to the number of symbols; Searching each node layer by layer from top to bottom along the DOM tree, so as to jointly determine whether the target node currently being searched belongs to the DOM node representing the body area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density; When it is determined that the target node currently being searched belongs to a DOM node representing the text area of ​​a web page, the DOM node representing the text area of ​​the web page is extracted as a candidate node corresponding to the text area.

2. The web page text area processing method according to claim 1, characterized in that: Get the DOM tree corresponding to the web page to be processed, including: Parse and render web page documents and resources corresponding to the web page to be processed through the browser or browser kernel; After the web page to be processed is parsed and rendered, DOM structure information, content information and visual information corresponding to the web page to be processed are obtained; A DOM tree corresponding to the web page to be processed is constructed according to the DOM structure information.

3. The web page text area processing method according to claim 2, characterized in that: Before parsing and rendering the web page document and resources corresponding to the web page to be processed, the web page text area processing method further includes: Receiving, via a browser or a browser kernel, a URL address corresponding to the web page to be processed; According to the target link corresponding to the web page to be processed, the web page document and the resource corresponding to the target link are obtained.

4. The web page text area processing method according to claim 1, characterized in that: After obtaining the DOM tree corresponding to the web page to be processed, the web page text area processing method further includes: deleting the DOM nodes irrelevant to the web page text information and all their descendant nodes from the DOM tree.

5. The web page text area processing method according to claim 1, characterized in that: Determining whether the target node currently being searched belongs to a DOM node representing a webpage text area according to the visual area ratio, the content length ratio and the symbol density includes: Determine whether the visual area ratio, the content length ratio, and the symbol density corresponding to the target node meet respective corresponding preset conditions; If the visual area ratio, the content length ratio and the symbol density all meet the corresponding preset conditions, it is determined that the target node currently being searched belongs to the DOM node representing the body area of ​​the web page.

6. The web page text area processing method according to claim 1, characterized in that: Search each node layer by layer along the DOM tree from top to bottom, including: Each node in the DOM tree is expressed using an xpath expression, where the xpath expression is expressed from top to bottom and from left to right in the DOM tree.

7. The method for processing the webpage text area according to claim 6, characterized in that: Extracting the DOM node representing the webpage text area as a candidate node corresponding to the text area includes: extracting the xpath expression corresponding to the DOM node representing the webpage text area as an xpath expression corresponding to the candidate node corresponding to the text area.

8. The method for processing the text area of ​​a web page according to any one of claims 1 to 7, characterized in that: The web page text area processing method further includes: determining whether the child nodes of each candidate node have been traversed.

9. A method for processing web page text information, characterized in that: include: Obtaining target candidate nodes corresponding to each text area in the web page to be processed, wherein the target candidate nodes are candidate nodes corresponding to the text areas obtained after the web page to be processed is processed by the web page text area processing method according to any one of claims 1 to 8; Clustering all child nodes under the target candidate node to obtain a first clustering result corresponding to the target candidate node; The child node corresponding to the largest cluster in the first clustering result is determined as a target data record, wherein the target data record is a data record containing text information.

10. The web page text information processing method according to claim 9, characterized in that: Before clustering all the child nodes under the target candidate node, the web page text information processing method further includes: Determine the similarity between any two child nodes of the target candidate node; Clustering all the child nodes under the target candidate node includes: clustering all the child nodes under the target candidate node according to the similarity.

11. The web page text information processing method according to claim 10, characterized in that: Determining the similarity between any two child nodes of the target candidate node includes: Determine the DOM structure similarity and visual information similarity between any two child nodes of the target candidate node; The similarity result between any two child nodes of the target candidate node is determined according to the weighted calculation of the DOM structure similarity and the visual information similarity, wherein the sum of the weight of the DOM structure similarity and the weight of the visual information similarity is 1.

12. The web page text information processing method according to claim 9, characterized in that: The web page text information processing method further includes: Extract the DOM subtree corresponding to each target data record; The DOM subtree is streamlined according to the visual information of each node in the DOM subtree.

13. The web page text information processing method according to claim 12, characterized in that: The DOM subtree is streamlined, including: searching from the leaf nodes of the DOM subtree from bottom to top, and when it is determined that all child element nodes of a block-level element node are row-level elements, merging the child element nodes of the block-level element node into the current block-level element node.

14. The web page text information processing method according to claim 12, characterized in that: The web page text information processing method further includes: extracting text content corresponding to the leaf nodes on the DOM subtree.

15. The web page text information processing method according to claim 14, characterized in that: The web page text information processing method further includes: deleting the noise information in the text content according to preset noise information.

16. The web page text information processing method according to claim 9, characterized in that: The web page text information processing method further includes: if it is found in the clustering result that each child node of the target candidate node is individually clustered, determining that the content in the area corresponding to the target candidate node does not belong to the target data record.

17. A web page text area processing system, characterized in that: include: A first acquisition module acquires a DOM tree corresponding to the web page to be processed, wherein a DOM node of the DOM tree contains visual information and content information corresponding to the DOM node; A first calculation module determines a visual area ratio corresponding to a target node in a DOM tree based on visual information of the target node, wherein the visual area ratio includes a ratio of a visual area of ​​the target node to a visual area of ​​the web page to be processed; A second calculation module determines, based on the content information of the target node in the DOM tree, a content length ratio and a symbol density corresponding to the target node, wherein the content length ratio includes a ratio of the sum of the text lengths of the target node and all its descendant nodes to the sum of the text lengths of all nodes on the web page to be processed, and the symbol density includes a ratio of the text length excluding links in the target node to the number of symbols; The first traversal module searches each node layer by layer from top to bottom along the DOM tree to jointly determine whether the target node currently being searched belongs to the DOM node representing the body area of ​​the web page according to the visual area ratio, the content length ratio and the symbol density; The first candidate module, when determining that the target node currently being searched belongs to a DOM node representing the body area of ​​a web page, extracts the DOM node representing the body area of ​​the web page as a candidate node corresponding to the body area.

18. A web page text information processing system, characterized in that: include: A second acquisition module, acquiring target candidate nodes corresponding to each text area in the web page to be processed, wherein the target candidate nodes are candidate nodes corresponding to the text area obtained after the web page to be processed is processed by the web page text area processing method according to any one of claims 1 to 8; A clustering module clusters all child nodes under the target candidate node; The second candidate module determines the child node corresponding to the largest cluster in the clustering result as the target data record, wherein the target data record is a data record containing text information.

19. An electronic device, characterized in that: include: at least one processor; And, a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute: the web page text area processing method described in any one of claims 1-8, or the web page text information processing method described in any one of claims 9-16.

20. A computer storage medium, characterized in that The computer storage medium stores computer executable instructions, and when the computer executable instructions are executed by the processor, the web page text area processing method described in any one of claims 1-8, or the web page text information processing method described in any one of claims 9-16 is executed.

Images