Webpage information extraction and classification method and device

Abstract

The application provides a webpage information extraction and classification method and device, and belongs to the technical field of artificial intelligence. The webpage information extraction and classification method comprises the following steps: converting the source code of a target webpage into a dom tree; processing each node of the dom tree to obtain four feature matrices, i.e., a text feature matrix, an Xpath feature matrix, a layout feature matrix and a visual feature matrix; inputting the four feature matrices into an encoding network respectively to obtain four representation vectors; performing feature fusion on the four representation vectors to obtain fused features; inputting the fused features into a classification network to obtain and store the classification result of an information unit of the target webpage, wherein the classification result comprises at least one of the following: a table, a form to be filled, a text unit that needs to be linked to a next webpage for display, a navigation bar or a display bar, pure text, an advertisement and useless information. The technical scheme of the application can improve the universality of the webpage information extraction and classification scheme.

Description

Technical Field

[0001] This invention relates to the field of artificial intelligence technology, and in particular to a method and apparatus for extracting and classifying web page information. Background Technology

[0002] With the development of mobile internet, facing a massive amount of information on the internet, how to quickly filter the web page information that users really need from a large number of websites is the key to content delivery. Therefore, web page information classification and extraction has become a hot research area and a new challenge.

[0003] Web page information classification and extraction involves traversing all relevant web pages under a root domain website using specific retrieval methods, extracting information from these pages, and classifying and storing it. Web page information extraction directly mines information from web pages. A web page information unit is a visual functional area displayed in an internet browser; each information unit has clear boundaries and is a coherent functional area. Web page information unit classification automatically segments and categorizes information units from a large number of web pages according to certain rules, thereby organizing the web pages in an orderly manner. This can improve information retrieval performance or store them as usable resources, thus increasing the utilization rate of network resources.

[0004] Existing methods for extracting web page information include: manual rule extraction based on the entire web page; regional segmentation based on tag density followed by topic information extraction; and segmentation based on nodes of the DOM tree followed by information extraction.

[0005] Among them, the manual rule extraction method based on the entire webpage involves a complex approach of classifying webpages at the granular level and then manually parsing the rules. This approach is time-consuming and labor-intensive, and it is difficult to extend to different webpages, resulting in poor versatility. The block-based method based on tag density is relatively coarse, leading to low information extraction accuracy throughout the process and reduced versatility. It is not suitable for displaying new webpages with increasingly rich elements. The classification algorithm based on DOM tree parsing cannot adapt to the increasing number of webpage formats and has poor versatility. Summary of the Invention

[0006] The technical problem to be solved by the present invention is to provide a method and apparatus for extracting and classifying web page information, which can improve the versatility of web page information extraction and classification schemes.

[0007] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0008] On one hand, embodiments of the present invention provide a method for extracting and classifying web page information, including:

[0009] Convert the source code of the target webpage into a DOM tree;

[0010] Each node of the DOM tree is processed to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix.

[0011] The four feature matrices are input into the encoding network respectively to obtain four representation vectors;

[0012] The four representation vectors are fused to obtain fused features. The fused features are then input into a classification network to obtain and store the classification results of the information units of the target webpage. The classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information.

[0013] In some embodiments, before converting the source code of the target webpage into a DOM tree, the method further includes:

[0014] Based on the root domain site of the webpage to be processed, the URL data corresponding to all sub-sites of the webpage to be processed is obtained through deep search, and the URL address of each sub-site and the content of its corresponding information unit are stored in the first table.

[0015] Traverse the first table to identify unparsed URLs and determine the webpage corresponding to the URL as the target webpage.

[0016] In some embodiments, before processing each node of the dom tree, the method further includes:

[0017] Remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

[0018] In some embodiments, each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the text feature of the node;

[0019] Each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the XPath feature of the node;

[0020] Each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the layout feature of the node.

[0021] Each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the visual feature of the node.

[0022] In some embodiments, obtaining the XPath feature matrix includes:

[0023] Obtain the XPath path corresponding to the node of the DOM tree, including:

[0024] Based on the source code, use the html.Xpath function to obtain all nodes;

[0025] Iterate through all nodes, set a flag, and increment the flag if the current path is contained in the next path. If the current path is not contained in the next path, add the current path as a leaf node path to the result list.

[0026] The result list is compared with the original path list. If a path in the result list is not contained in any path in the original path list, the path is determined to be a leaf node path.

[0027] In some embodiments, the step of inputting the four feature matrices into the encoding network to obtain four representation vectors includes at least one of the following:

[0028] The text feature matrix is ​​input into the multilayer self-attention mechanism network BERT to obtain the first representation vector;

[0029] The XPath feature matrix is ​​input into a convolutional neural network to obtain a second representation vector;

[0030] The layout feature matrix is ​​input into a convolutional neural network to obtain a third representation vector;

[0031] The visual feature matrix is ​​input into the FCOS detection network to obtain the fourth representation vector.

[0032] This invention also provides a web page information extraction and classification device, comprising:

[0033] The conversion module is used to convert the source code of the target webpage into a DOM tree;

[0034] The processing module is used to process each node of the DOM tree to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix.

[0035] The encoding module is used to input the four feature matrices into the encoding network respectively to obtain four representation vectors;

[0036] The classification module is used to fuse the four representation vectors to obtain fused features, input the fused features into the classification network, obtain and store the classification results of the information units of the target webpage, and the classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information.

[0037] In some embodiments, the apparatus further includes:

[0038] The search module is used to obtain the URL data corresponding to all sub-sites of the webpage to be processed through deep search based on the root domain site of the webpage to be processed, and store the URL address of each sub-site and the content of its corresponding information unit in the first table.

[0039] The determination module is used to traverse the first table, determine the URL address that has not been parsed, and determine the webpage corresponding to the URL address as the target webpage.

[0040] In some embodiments, the apparatus further includes:

[0041] The removal module is used to remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

[0042] In some embodiments, each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the text feature of the node;

[0043] Each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the XPath feature of the node;

[0044] Each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the layout feature of the node.

[0045] Each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the visual feature of the node.

[0046] In some embodiments, the encoding module is specifically configured to perform at least one of the following:

[0047] The text feature matrix is ​​input into the multilayer self-attention mechanism network BERT to obtain the first representation vector;

[0048] The XPath feature matrix is ​​input into a convolutional neural network to obtain a second representation vector;

[0049] The layout feature matrix is ​​input into a convolutional neural network to obtain a third representation vector;

[0050] The visual feature matrix is ​​input into the FCOS detection network to obtain the fourth representation vector.

[0051] This invention also provides a web page information extraction and classification device, including a memory, a processor, and a computer program stored in the memory and executable on the processor; when the processor executes the program, it implements the web page information extraction and classification method as described above.

[0052] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the web page information extraction and classification method described above.

[0053] The embodiments of the present invention have the following beneficial effects:

[0054] The above scheme obtains multi-dimensional features of the target webpage: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix. In view of the variability of webpage characteristics, the multi-dimensional features are fused to improve the universality of webpage information classification. In addition, the information units of a large number of webpages are uniformly classified into fixed information unit categories to further improve the universality of webpage information classification. Attached Figure Description

[0055] Figure 1 This is a flowchart illustrating the webpage information extraction and classification method according to an embodiment of the present invention;

[0056] Figure 2 A schematic diagram illustrating the process of establishing a URL storage table according to an embodiment of the present invention;

[0057] Figure 3 This is a schematic diagram illustrating the process of classifying web page information according to an embodiment of the present invention;

[0058] Figure 4 A schematic diagram illustrating the generation of feature vectors in an embodiment of the present invention;

[0059] Figure 5 This is a schematic diagram illustrating the convolution process of a convolutional neural network according to an embodiment of the present invention.

[0060] Figure 6 This is a schematic diagram of the web page information extraction and classification device according to an embodiment of the present invention;

[0061] Figure 7 This is a schematic diagram of the web page information extraction and classification device according to an embodiment of the present invention. Detailed Implementation

[0062] To make the technical problems, technical solutions and advantages of the embodiments of the present invention clearer, a detailed description will be given below in conjunction with the accompanying drawings and specific embodiments.

[0063] Existing methods for extracting web page information include: extracting web page information based on rule-based generation; dividing regions into blocks based on tag density and then extracting topic information; and extracting region block information after clustering based on node features of the DOM tree.

[0064] In the rule-based webpage information extraction method, technicians start from the website, using their knowledge of the webpage source code language and extensively reviewing webpages in the dataset to summarize the patterns and extraction rules for key information. This method requires using complex manually generated rules to match repetitive elements in the webpage body. While manual rules can improve the accuracy of information extraction, they require significant time and manpower for optimization and selection across different webpage sets. Furthermore, the universality of manual rules is limited, making it difficult to extend to different webpages and resulting in poor versatility.

[0065] The method of dividing regions into blocks based on tag density and then extracting thematic information relies on the tag density characteristics of text, tables, etc. on web pages. For new web pages with rich display elements and methods, this will result in fragmented block results, poor versatility, and is not suitable for displaying new web pages with increasingly rich elements.

[0066] This method, which extracts region block information by clustering node features based on DOM trees, uses methods such as path length and common sequence length between nodes to cluster and divide the data into blocks, then extracts information from each region block. However, this method relies on complex DOM tree node feature analysis (including relationships between nodes and node characteristics), and cannot adapt to the increasing variety of web page formats.

[0067] This invention provides a method and apparatus for web page information extraction and classification, which can improve the versatility of web page information extraction and classification schemes.

[0068] Embodiments of the present invention provide a method for extracting and classifying web page information, such as... Figure 1 As shown, it includes:

[0069] Step 101: Convert the source code of the target webpage into a DOM tree;

[0070] Step 102: Process each node of the DOM tree to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix;

[0071] Step 103: Input the four feature matrices into the encoding network respectively to obtain four representation vectors;

[0072] Step 104: Perform feature fusion on the four representation vectors to obtain fused features. Input the fused features into a classification network to obtain and store the classification results of the information units of the target webpage. The classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information.

[0073] In this embodiment, multi-dimensional features of the target webpage are obtained: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix. Given the variability of webpage characteristics, these multi-dimensional features are fused to improve the versatility of webpage information classification. Furthermore, a large number of webpage information units are uniformly classified into fixed information unit categories, further enhancing the versatility of webpage information classification.

[0074] In some embodiments, before converting the source code of the target webpage into a DOM tree, the method further includes:

[0075] Based on the root domain site of the webpage to be processed, the URL (Uniform Resource Locator) data corresponding to all sub-sites of the webpage to be processed is obtained through deep search, and the URL address of each sub-site and the content of its corresponding information unit are stored in the first table.

[0076] Traverse the first table to identify unparsed URLs and determine the webpage corresponding to the URL as the target webpage.

[0077] like Figure 2 As shown, creating the first table specifically includes the following steps:

[0078] Step 201: Read the root domain site of the webpage to be processed;

[0079] Step 202: Obtain the URL data corresponding to all subsites of the webpage to be processed through deep search;

[0080] The webpage to be processed can include multiple sub-links, and the sub-site is the webpage corresponding to the sub-link.

[0081] Step 203: Create and store the first table (i.e., the URL storage table).

[0082] The first table contains the URLs of subsites and their corresponding information units, stored in the format U = {Subsite URL: [Information Unit Storage Table ID_Corresponding Information]}. The content of the first table can be updated at a specified period. If a key value corresponding to a URL address in the first table is empty, it means that the URL address has not been resolved, and information extraction and classification of the webpage corresponding to the URL address are required. An empty key value for a URL address means that the first table only stores the URL address and does not store the content of the information unit corresponding to the URL address.

[0083] like Figure 3 As shown, classifying webpage information includes the following steps:

[0084] Step 301: Read the URL storage table;

[0085] Step 302: Traverse the URL storage table;

[0086] Step 303: Determine if the value corresponding to the URL address is empty. If yes, proceed to step 304; otherwise, proceed to step 302.

[0087] By determining whether the value corresponding to the URL address is empty, unparsed URL addresses can be identified, and the webpages corresponding to these unparsed URL addresses can be identified as the target webpages. This embodiment avoids repetitive webpage parsing processes and improves the efficiency of webpage information extraction by traversing all webpages.

[0088] Step 304: Perform feature extraction, convert the source code of the target webpage into a DOM tree, process each valid node of the DOM tree as a granularity, and obtain the text features, XPath features, layout features, and visual features corresponding to each node.

[0089] In some embodiments, before processing each node of the dom tree, the method further includes:

[0090] Remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

[0091] In this embodiment, the source code of the target webpage is converted into a DOM tree. The DOM tree treats the webpage as a tree structure, through which all nodes can be accessed, their content can be modified or deleted, and new elements can be created. This node tree displays the collection of nodes and the relationships between them. The tree starts from the root node and then branches out to text nodes at the lowest level. In this embodiment, an HTML parser can be used to convert the source code of each webpage into a DOM tree; then, invalid information is removed from the webpage source code: that is, tags that do not affect the page structure are removed, including JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content. Specifically, the page tag framework is obtained based on the standard HTML tag set, and tags are removed using regular expressions as shown in Table 1. Removing invalid information from the source code of the target webpage avoids subsequent processing of invalid information and improves the efficiency of webpage information extraction.

[0092] Table 1

[0093] Regular expressions Invalid information (?is)<!--[^>]*--> HTML tag comments (?is)<\s*link[^>]*> Hyperlink tags (?is)<input.*?> Introducing tags (?is)<select.*?>.*? select tag

[0094] A webpage feature extractor can represent a target webpage as a four-dimensional feature matrix, designated V1, V2, V3, and V4, containing text features, layout features, XPath features, and visual features, respectively. Each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the node's text feature. Similarly, each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the node's XPath feature. Likewise, each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the node's layout feature. Finally, each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the node's visual feature.

[0095] Among them, visual features include visual information features, which are various characteristics corresponding to page information segments, such as color, background color, tags, parent tags, size, structural code, position, links, server-actions, client-actions, and text; on the other hand, visual features also include visual morphological features, which are the visual results of the page.

[0096] Obtaining the XPath feature matrix includes obtaining the XPath paths corresponding to the nodes of the DOM tree. The XPath of a node has the following characteristics: there is containment between paths; the path of a leaf node necessarily contains the path between the previous leaf node and the next leaf node. In this embodiment, based on the source code, the html.XPath function is first used to obtain all nodes. The nodes are traversed, and a flag is set. If the current path is contained by the next path, the index is incremented by 1 until a path that is not contained is encountered; this path is a leaf node path and is added to the result list. After verification and comparison with the original path list, if the path list does not show containment, then all paths are leaf node paths and are saved under XPath path. This step is a verification step, that is, verifying that the paths in the obtained result list are indeed leaf node paths. The specific steps are: traversing the obtained result path list: for example, if the current traversal yields a path i, path i is compared with the original path list. If path i is not contained by any path in the original path set, then path i is determined to be a leaf node path.

[0097] The resulting node XPath paths are shown below:

[0098] / html / body / div[1] / div[2] / a[1]

[0099] / html / body / div[1] / div[2] / a[2]

[0100] / html / body / div[1] / div[2] / a[3]

[0101] / html / body / div[1] / div[2] / a[4]

[0102] / html / body / div[1] / div[2] / a[5]

[0103] / html / body / div[1] / div[2] / a[6]

[0104] / html / body / div[1] / div[2] / a[7]

[0105] / html / body / div[1] / div[2] / a[8]

[0106] / html / body / div[1] / div[2] / a[9].

[0107] Step 305: Input the four feature matrices into the encoding network respectively;

[0108] like Figure 4As shown, the text feature matrix can be input into the BERT network to obtain the first feature vector, the XPath feature matrix can be input into the convolutional neural network to obtain the second feature vector, the layout matrix can be input into the convolutional neural network to obtain the third feature vector, and the visual feature matrix can be input into the FCOS network to obtain the fourth feature vector.

[0109] Because XPath paths have an inclusive property, most XPath paths have similar meanings. Therefore, one-hot encoding can be used to directly initialize the XPath path as an XPath feature. One-hot encoding, also known as one-bit valid encoding, mainly uses an N-bit state register to encode N states. Each state has an independent register bit, and only one bit is valid at any given time.

[0110] The layout corresponding to a node has a limited number of categories due to global influence. Therefore, we can directly use category features and then feed the layout features into a convolutional neural network for encoding.

[0111] The network encoding XPath features and layout features consists of N layers of one-dimensional convolutions. If a webpage's source code has 500 complete node text segments, then the feature representation dimension of that webpage is (500, N). The one-dimensional convolution direction is fixed in one direction, specified as N here. The convolution principle is as follows: Figure 5 As shown, the text segment of the webpage source code is convolved with the convolution kernel to obtain the feature vector (feature map).

[0112] For text features, a multi-layer self-attention network BERT can be used to map the text segments in the webpage source code into a 768-dimensional representation vector.

[0113] Visual features include specific visual information (such as font, color, text density, font size, font color, whether it is bold, and whether it has an underline). In addition, visual features can also incorporate the presentation of the webpage containing the node, mapping the webpage to an image. From a visual perspective, the FCOS detection network can be used to encode visual features to understand the webpage's form.

[0114] Step 306: Perform feature fusion on the four representation vectors to obtain fused features;

[0115] Multi-dimensional feature fusion can be achieved using a three-layer neural network mapping. The four feature vectors are input into the three-layer neural network to obtain fused features. Nodes with the same visual information on a webpage will have similarities in terms of visual appearance, text, and layout. However, this characteristic pattern will change in complex webpages. Therefore, this embodiment fuses multi-dimensional features for information classification.

[0116] Step 307: Classify.

[0117] The fused feature vectors are input into the classification layer (softmax network) to obtain the classification results of the web page information units corresponding to the nodes. The classification results of the web page information units include the following categories:

[0118] (1) Table type: This type of information unit is a table containing product descriptions, etc. The storage format of this type of information unit is a list, and the elements are product descriptions in dictionary form {product name: product description};

[0119] (2) Fillable Form (Form) Class: Form class information units are fillable forms on a webpage that contain triggers. Filling this type of information unit with information will trigger a hidden server application. This type of information unit needs to be labeled with a text description of the Form when stored. The storage format for this type of information unit is a list, with elements in dictionary form: {form text description: trigger function module};

[0120] (3) The text unit (Notation) class that needs to link to the next webpage stores the hyperlink explantation of that webpage and the detailed text content following the hyperlink. If the value corresponding to the URL address in the URL storage table is Notation, it indicates that the URL address has been resolved. The storage format of this type of information unit is a list, and the elements are in dictionary form {notation: hyperlink};

[0121] (4) Navigation bar or display bar (Menu): When storing the menu, the navigation bar text description is stored as a foreign key, which links to the corresponding parsed storage table of the hyperlinks after navigation. The storage format of this type of information unit is a list, and the elements are in dictionary form {navigation text: navigation link content};

[0122] (5) Plain Text: Plain text information units without any hyperlinks. These information units are stored as lists, with each element being text content.

[0123] (6) Other categories, including advertisements and irrelevant information. No specific content is stored; only the relevant titles are stored.

[0124] This embodiment categorizes webpage information units, classifying a large amount of webpage information into fixed information unit categories. This simplifies the information extraction process and improves the versatility of the information extraction method by utilizing the repetitive patterns of these information units. This embodiment presents a webpage information extraction method that does not require complex information extraction rule learning, enabling information extraction from the entire website, starting from the root site, once a website is given.

[0125] This embodiment avoids the need for complex rule analysis of webpage structure required for classification based on webpage granularity. Instead, it proposes a classification scheme based on the function of information units, which fundamentally optimizes the classification results of webpage information units and improves the effectiveness and universality of information classification.

[0126] This embodiment addresses the flexible and ever-changing nature of web pages by introducing deep search and a first table of associations to handle massive amounts of web pages. Simultaneously, the separation of feature extraction and information classification enhances the scalability of the entire solution, improving its flexibility and versatility.

[0127] This invention also provides a web page information extraction and classification device, such as... Figure 6 As shown, it includes:

[0128] The conversion module 41 is used to convert the source code of the target webpage into a DOM tree;

[0129] Processing module 42 is used to process each node of the DOM tree to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix and visual feature matrix;

[0130] Encoding module 43 is used to input the four feature matrices into the encoding network respectively to obtain four representation vectors;

[0131] The classification module 44 is used to fuse the four representation vectors to obtain fused features, input the fused features into the classification network, obtain and store the classification results of the information units of the target webpage, and the classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information.

[0132] In this embodiment, multi-dimensional features of the target webpage are obtained: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix. Given the variability of webpage characteristics, these multi-dimensional features are fused to improve the versatility of webpage information classification. Furthermore, a large number of webpage information units are uniformly classified into fixed information unit categories, further enhancing the versatility of webpage information classification.

[0133] In some embodiments, the apparatus further includes:

[0134] The search module is used to obtain the URL data corresponding to all sub-sites of the webpage to be processed through deep search based on the root domain site of the webpage to be processed, and store the URL address of each sub-site and the content of its corresponding information unit in the first table.

[0135] The determination module is used to traverse the first table, determine the URL address that has not been parsed, and determine the webpage corresponding to the URL address as the target webpage.

[0136] In some embodiments, the apparatus further includes:

[0137] The removal module is used to remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

[0138] In some embodiments, each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the text feature of the node;

[0139] Each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the XPath feature of the node;

[0140] Each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the layout feature of the node.

[0141] Each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the visual feature of the node.

[0142] In some embodiments, the encoding module 43 is specifically configured to perform at least one of the following:

[0143] The text feature matrix is ​​input into the multilayer self-attention mechanism network BERT to obtain the first representation vector;

[0144] The XPath feature matrix is ​​input into a convolutional neural network to obtain a second representation vector;

[0145] The layout feature matrix is ​​input into a convolutional neural network to obtain a third representation vector;

[0146] The visual feature matrix is ​​input into the FCOS detection network to obtain the fourth representation vector.

[0147] This invention also provides a web page information extraction and classification device, such as... Figure 7 As shown, it includes a memory 31, a processor 32, and a computer program stored on the memory 31 and executable on the processor 32; when the processor 32 executes the program, it implements the web page information extraction and classification method as described above.

[0148] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps in the web page information extraction and classification method described above.

[0149] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage, or any other non-transferable medium that can be used to store information accessible to the computer-readable terminal device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0150] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for extracting and classifying web page information, characterized in that, include: Convert the source code of the target webpage into a DOM tree; Each node of the DOM tree is processed to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix. The four feature matrices are input into the encoding network respectively to obtain four representation vectors; The four representation vectors are fused to obtain fused features. The fused features are then input into a classification network to obtain and store the classification results of the information units of the target webpage. The classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information. Each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the text feature of the node; Each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the XPath feature of the node; Each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the layout feature of the node. Each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the visual feature of the node; The step of inputting the four feature matrices into the encoding network to obtain four representation vectors includes at least one of the following: The text feature matrix is ​​input into the multilayer self-attention mechanism network BERT to obtain the first representation vector; The XPath feature matrix is ​​input into a convolutional neural network to obtain a second representation vector; The layout feature matrix is ​​input into a convolutional neural network to obtain a third representation vector; The visual feature matrix is ​​input into the FCOS detection network to obtain the fourth representation vector.

2. The webpage information extraction and classification method according to claim 1, characterized in that, Before converting the source code of the target webpage into a DOM tree, the method further includes: Based on the root domain site of the webpage to be processed, the URL data corresponding to all sub-sites of the webpage to be processed is obtained through deep search, and the URL address of each sub-site and the content of its corresponding information unit are stored in the first table. Traverse the first table to identify unparsed URLs and determine the webpages corresponding to those URLs as the target webpages.

3. The webpage information extraction and classification method according to claim 1, characterized in that, Before processing each node of the dom tree, the method further includes: Remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

4. The webpage information extraction and classification method according to claim 1, characterized in that, The XPath feature matrix is ​​obtained by: Obtain the XPath path corresponding to the node of the DOM tree, including: Based on the source code, use the html.Xpath function to obtain all nodes; Iterate through all nodes, set a flag, and increment the flag if the current path is contained in the next path. If the current path is not contained in the next path, add the current path as a leaf node path to the result list. The result list is compared with the original path list. If a path in the result list is not contained in any path in the original path list, the path is determined to be a leaf node path.

5. A web page information extraction and classification device, characterized in that, include: The conversion module is used to convert the source code of the target webpage into a DOM tree; The processing module is used to process each node of the DOM tree to obtain four feature matrices: text feature matrix, XPath feature matrix, layout feature matrix, and visual feature matrix. The encoding module is used to input the four feature matrices into the encoding network respectively to obtain four representation vectors; The classification module is used to fuse the four representation vectors to obtain fused features, input the fused features into the classification network, obtain and store the classification results of the information units of the target webpage, and the classification results include at least one of the following: tables; forms that need to be filled in; text units that need to be linked to the next webpage; navigation bars or display bars; plain text; advertisements and useless information; Each row of the text feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the text feature of the node; Each row of the XPath feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the XPath feature of the node; Each row of the layout feature matrix corresponds to a node in the source code, and each column corresponds to the feature value of the layout feature of the node. Each row of the visual feature matrix corresponds to a node in the source code, and each column corresponds to the characteristic value of the visual feature of the node; The encoding module is specifically used to perform at least one of the following: The text feature matrix is ​​input into the multilayer self-attention mechanism network BERT to obtain the first representation vector; The XPath feature matrix is ​​input into a convolutional neural network to obtain a second representation vector; The layout feature matrix is ​​input into a convolutional neural network to obtain a third representation vector; The visual feature matrix is ​​input into the FCOS detection network to obtain the fourth representation vector.

6. The webpage information extraction and classification device according to claim 5, characterized in that, The device further includes: The search module is used to obtain the URL data corresponding to all sub-sites of the webpage to be processed through deep search based on the root domain site of the webpage to be processed, and store the URL address of each sub-site and the content of its corresponding information unit in the first table. The determination module is used to traverse the first table, determine the URL address that has not been parsed, and determine the webpage corresponding to the URL address as the target webpage.

7. The webpage information extraction and classification device according to claim 5, characterized in that, The device further includes: The removal module is used to remove invalid information from the source code of the target webpage. The invalid information includes at least one of the following: JavaScript tags, HTML comment tags, stylesheet import tags, and tags that do not contain any text content.

8. A web page information extraction and classification device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor; characterized in that, When the processor executes the program, it implements the web page information extraction and classification method as described in any one of claims 1-5.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by the processor, it implements the steps in the web page information extraction and classification method as described in any one of claims 1-5.

Images