Webpage information extraction method and device, electronic equipment and storage medium

Abstract

Embodiments of the present application provide a webpage information extraction method and device, electronic equipment and storage medium. The method comprises: acquiring all webpages accessed by a user; filtering all the webpages according to page update time and page entry and exit port information of the webpages to obtain filtered webpages; constructing a webpage layout tree according to webpage pointing relationships corresponding to the filtered webpages; and extracting target webpage information of a target webpage label from the webpage layout tree. The embodiments of the present application can improve webpage screening accuracy and effectively extract important information in a webpage.

Description

Technical Field

[0001] This application relates to the field of web page information processing technology, and in particular to a web page information extraction method, apparatus, electronic device and storage medium. Background Technology

[0002] With the continuous development of science and technology, the application of network technology is becoming increasingly widespread. As society continues to progress informatization, the amount of data shared via the internet is also constantly increasing. Filtering and extracting the necessary important information from this massive amount of shared data is a challenge. Therefore, obtaining key web pages and extracting important information from them are urgent problems that need to be solved. Summary of the Invention

[0003] The technical problem to be solved by the embodiments of this application is to provide a web page information extraction method, apparatus, electronic device and storage medium to effectively filter useless web pages and improve the accuracy of web page screening. By establishing a web page layout tree, important information in the web page can be effectively extracted.

[0004] In a first aspect, embodiments of this application provide a method for extracting web page information, including:

[0005] Get all the web pages visited by the user;

[0006] The web pages are filtered based on their update time and inbound / outbound port information to obtain the filtered web pages.

[0007] Based on the webpage pointing relationships corresponding to the filtered webpages, construct a webpage layout tree;

[0008] Extract the target webpage information from the target webpage tags in the webpage layout tree.

[0009] Optionally, obtaining all web pages visited by the user includes:

[0010] The set of URLs accessed by the user is crawled using web crawling technology;

[0011] Based on the URLs in the URL set, determine all the web pages accessed by the user.

[0012] Optionally, after determining all the web pages visited by the user based on the URLs in the URL set, the method further includes:

[0013] Based on the redirection information of the URLs in the URL set, determine the webpage pointing relationship between all the webpages;

[0014] Based on all the web pages and the web page pointing relationships, construct a web page pointing relationship dataset.

[0015] Optionally, the step of filtering all web pages based on their update time and inbound / outbound port information to obtain filtered web pages includes:

[0016] The T-HITS algorithm is used to determine the page weight of each page based on its page update time and the number of inbound and outbound ports.

[0017] Based on the page layout information and URL call information of all the web pages, a score is assigned to each web page to obtain a web page score value.

[0018] Based on the webpage weight and the webpage rating, the webpage weight score of all webpages is calculated;

[0019] The web pages are filtered based on their weight scores to obtain the filtered web pages.

[0020] Optionally, the step of filtering all web pages based on the web page weight score to obtain filtered web pages includes:

[0021] The web pages whose weight scores are greater than a weight score threshold are obtained from all the web pages, and these web pages with weight scores greater than the weight score threshold are used as the filtered web pages.

[0022] Optionally, constructing a webpage layout tree based on the webpage pointing relationships corresponding to the filtered webpages includes:

[0023] Based on the webpage weight scores in descending order, the webpage code of the filtered webpages is parsed sequentially to obtain the DOM node tree corresponding to each webpage in the filtered webpages;

[0024] Based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimensions, construct the webpage layout tree corresponding to the filtered webpage.

[0025] Optionally, constructing the webpage layout tree corresponding to the filtered webpages based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimensions includes:

[0026] Based on the webpage pointing relationship, construct an initial webpage layout tree with the filtered webpages as parent nodes;

[0027] Based on the DOM node tree and the preset webpage splitting dimensions, extract the target webpage feature information within the filtered webpage;

[0028] Based on the target webpage feature information, construct the child nodes of the parent node to obtain the webpage layout tree.

[0029] Secondly, embodiments of this application provide a web page information extraction device, comprising:

[0030] The "All Webpages Retrieval" module is used to retrieve all webpages accessed by the user.

[0031] The webpage filtering module is used to filter all webpages based on their update time and inbound / outbound port information to obtain filtered webpages.

[0032] The webpage layout tree construction module is used to construct a webpage layout tree based on the webpage pointing relationship corresponding to the filtered webpages;

[0033] The target webpage information extraction module is used to extract target webpage information of target webpage tags from the webpage layout tree.

[0034] Optionally, the all webpage acquisition modules include:

[0035] The URL set crawling unit is used to crawl the set of URLs accessed by the user based on web crawling technology;

[0036] All webpage acquisition units are used to determine all webpages accessed by the user based on the URLs in the URL set.

[0037] Optionally, the device further includes:

[0038] The webpage pointing relationship determination module is used to determine the webpage pointing relationship between all webpages based on the redirection information of the URLs in the URL set;

[0039] The pointer dataset construction module is used to construct a webpage pointer relationship dataset based on all the webpages and the webpage pointer relationships.

[0040] Optionally, the webpage filtering module includes:

[0041] The webpage weight determination unit is used to call the T-HITS algorithm to determine the webpage weight of each webpage based on the page update time and the number of inbound and outbound ports of each webpage.

[0042] The webpage rating unit is used to rate all webpages based on their page layout information and URL call information, and obtain the webpage rating values ​​of all webpages.

[0043] The webpage weight score calculation unit is used to calculate the webpage weight score of all webpages based on the webpage weight and the webpage score.

[0044] The webpage filtering unit is used to filter all webpages according to the webpage weight score to obtain filtered webpages.

[0045] Optionally, the webpage filtering and acquisition unit includes:

[0046] The webpage filtering acquisition subunit is used to acquire webpages whose webpage weight scores are greater than a weight score threshold among all webpages, and to use the webpages whose webpage weight scores are greater than the weight score threshold as the filtered webpages.

[0047] Optionally, the webpage layout tree construction module includes:

[0048] The DOM node tree acquisition unit is used to parse the web page code of the filtered web pages in descending order of the web page weight scores to obtain the DOM node tree corresponding to each web page in the filtered web pages.

[0049] The webpage layout tree construction unit is used to construct the webpage layout tree corresponding to the filtered webpage based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimension.

[0050] Optionally, the webpage layout tree construction unit includes:

[0051] The initial layout tree construction sub-unit is used to construct an initial webpage layout tree with the filtered webpages as parent nodes according to the webpage pointing relationship;

[0052] The target webpage feature extraction subunit is used to extract target webpage feature information within the filtered webpage based on the DOM node tree and the preset webpage splitting dimensions.

[0053] The webpage layout tree construction subunit is used to construct the child nodes of the parent node based on the target webpage feature information to obtain the webpage layout tree.

[0054] Thirdly, embodiments of this application provide an electronic device, including:

[0055] A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the web page information extraction method described above.

[0056] Fourthly, embodiments of this application provide a computer-readable storage medium that, when the instructions in the storage medium are executed by a processor of an electronic device, enables the electronic device to perform the web page information extraction method described in any of the preceding claims.

[0057] Compared with the prior art, the embodiments of this application have the following advantages:

[0058] In this embodiment, all web pages accessed by the user are obtained, and all web pages are filtered based on their update time and inbound / outbound port information to obtain filtered web pages. A web page layout tree is then constructed based on the corresponding web page relationships of the filtered web pages, and target web page information (target web page tags) is extracted from the layout tree. This embodiment effectively filters useless web pages by combining page update time and inbound / outbound port information, thereby improving the accuracy of web page filtering. Furthermore, the establishment of a web page layout tree allows for the effective extraction of important information within the web pages.

[0059] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description

[0060] Figure 1 A flowchart illustrating the steps of a webpage information extraction method provided in this application embodiment;

[0061] Figure 2 A flowchart illustrating the steps of a webpage acquisition method provided in this application embodiment;

[0062] Figure 3 A flowchart illustrating the steps of a method for constructing a webpage-to-relationship dataset, as provided in this application embodiment;

[0063] Figure 4 A flowchart illustrating the steps of a method for obtaining filtered web pages provided in this application embodiment;

[0064] Figure 5 A flowchart illustrating the steps of a webpage layout tree construction method provided in this application embodiment;

[0065] Figure 6 A flowchart illustrating the steps of another webpage layout tree construction method provided in this application embodiment;

[0066] Figure 7 This is a schematic diagram of the structure of a web page information extraction device provided in an embodiment of this application;

[0067] Figure 8 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0068] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0069] The terminology used in the embodiments of this application is for the purpose of describing particular embodiments only and is not intended to be limiting of this application. The singular forms “a,” “the,” and “the” used in the embodiments of this application and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise.

[0070] Reference Figure 1 The flowchart illustrates the steps of a webpage information extraction method provided in an embodiment of this application, such as... Figure 1 As shown, this webpage information extraction method may include the following steps:

[0071] Step 101: Obtain all the web pages visited by the user.

[0072] The embodiments of this application can be applied to scenarios where target webpage information is extracted from a webpage layout tree constructed based on the remaining pages after filtering out useless pages from the user's access page.

[0073] All web pages refer to all web pages accessed by a user within a certain period of time. In this example, the web pages accessed by the user can be HTML web pages, web pages, etc. Specifically, the type of web pages accessed by the user can be determined according to the actual situation, and this embodiment does not impose any restrictions on this.

[0074] When a user accesses the internet, the web pages they visit can be monitored to obtain all the web pages they access. In practice, this can be achieved using web crawling technology to extract the URLs visited by the user and retrieve all the web pages they access. This process can be combined with... Figure 2 The following is a detailed description.

[0075] Reference Figure 2 The diagram illustrates a flowchart of a webpage acquisition method provided in an embodiment of this application. Figure 2 As shown, the method for obtaining the webpage may include steps 201 and 202.

[0076] Step 201: Use web crawling technology to crawl the set of URLs accessed by the user.

[0077] In this embodiment, a web crawler is a program that automatically retrieves web page content and is an important component of a search engine.

[0078] When a user visits web pages, the set of URLs (Uniform Resource Locators) accessed by the user can be retrieved using web crawler technology. Specifically, the set of URLs accessed by the user can be retrieved using a web crawler program pre-installed on the system.

[0079] After obtaining the set of URLs accessed by the user using web crawling technology, proceed to step 202.

[0080] Step 202: Determine all web pages accessed by the user based on the URLs in the URL set.

[0081] After obtaining a set of URLs accessed by the user using web crawling technology, all web pages accessed by the user can be determined based on the URLs in the set.

[0082] This application embodiment uses web crawler technology to obtain all web pages accessed by the user, which can improve the accuracy of web page crawling.

[0083] In this embodiment, after obtaining all the web pages visited by the user, a web page pointing relationship dataset can be built based on all the web pages. Specifically, it can be combined with... Figure 3 The following is a detailed description.

[0084] Reference Figure 3 The flowchart illustrates the steps of a method for constructing a webpage-to-relationship dataset according to an embodiment of this application. Figure 3 As shown, the webpage points to a relational dataset construction method that may include steps 301 and 302.

[0085] Step 301: Determine the webpage pointing relationship between all webpages based on the redirection information of the URLs in the URL set.

[0086] In this embodiment, the webpage pointing relationship refers to the pointing relationship between all the webpages accessed by the user. For example, during the user's access process, if the user jumps from webpage A to webpage B, it can be indicated that webpage A points to webpage B, etc.

[0087] It is understood that the above examples are merely examples listed to better understand the technical solutions of the embodiments of this application, and are not intended to be the only limitation on the embodiments.

[0088] After obtaining the set of URLs accessed by the user using web crawling technology, the web page linking relationship between all web pages can be determined based on the redirection information of the URLs in the URL set.

[0089] After determining the webpage pointing relationships between all webpages based on the URL redirection information in the URL set, proceed to step 302.

[0090] Step 302: Construct a webpage pointing relationship dataset based on all the webpages and the webpage pointing relationships.

[0091] After determining the webpage linking relationships between all webpages based on the URL redirection information in the URL set, a webpage linking relationship dataset can be constructed based on all webpages and their linking relationships. This dataset contains the linking relationships between all webpages. For example, if all webpages include: webpage A, webpage B, webpage C, webpage D, webpage E, and webpage F, the linking relationships are: webpage A—webpage B—webpage C, webpage A—webpage D, webpage A—webpage E—webpage F, etc.

[0092] It is understood that the above examples are merely examples listed to better understand the technical solutions of the embodiments of this application, and are not intended to be the only limitation on the embodiments.

[0093] This application embodiment establishes a webpage pointing relationship dataset, which can instruct the subsequent construction of the webpage layout tree to be carried out according to the pointing relationship, thereby accurately understanding the relationship between webpages and facilitating the extraction of target webpage information.

[0094] After obtaining all the web pages visited by the user, proceed to step 102.

[0095] Step 102: Filter all web pages according to their page update time and page inbound / outbound port information to obtain filtered web pages.

[0096] After obtaining all the web pages visited by the user, all web pages can be filtered based on their update time and inbound / outbound port information, with the remaining pages being selected as the final filtered pages. The implementation process for web page filtering can be combined with... Figure 4 The following is a detailed description.

[0097] Reference Figure 4 The flowchart illustrates the steps of a webpage filtering and acquisition method provided in an embodiment of this application. Figure 4 As shown, the method for obtaining filtered web pages may include steps 401, 402, 403, and 404.

[0098] Step 401: Use the T-HITS algorithm to determine the page weight of each page based on its page update time and the number of inbound and outbound ports.

[0099] In this embodiment, the page update time refers to the time when each page was last updated.

[0100] Page ingress / egress ports refer to the number of tags within a target page that can be navigated to from a specific tag on another page. In this example, each page can have only one page ingress / egress port, or it can have multiple page ingress / egress ports.

[0101] After obtaining all the web pages visited by the user, the update time and the number of inbound and outbound ports for each page can be obtained. Then, based on these data, appropriate page weights are assigned to each page. Specifically, if a page's update time is relatively recent and its number of inbound and outbound ports is high, a higher weight can be assigned to that page. Conversely, if a page's update time is relatively distant and its number of inbound and outbound ports is low, a lower weight can be assigned to that page, and so on.

[0102] In the specific implementation, a weight value can be set in advance for different numbers of days between the page update time and the current time, and a weight value can be assigned to different numbers of inbound and outbound ports for the page. Then, the two weight values ​​corresponding to the page update time and the number of inbound and outbound ports for all web pages can be added together to obtain the page weight of each web page in all web pages.

[0103] It is understood that the above-described methods of assigning web page weights are merely examples for better understanding the technical solutions of the embodiments of this application. In specific implementations, other methods can also be used to assign web page weights to all web pages based on the page update time and the number of inbound and outbound ports. Specifically, this can be determined according to business needs, and this embodiment does not impose any restrictions on this.

[0104] The HITS algorithm is a web link analysis algorithm. However, the traditional HITS algorithm is prone to ignoring new pages and is susceptible to spam links. Therefore, this embodiment improves the traditional HITS algorithm, resulting in the T-HITS algorithm. This T-HITS algorithm determines the page weight of each webpage based on its update time and the number of inbound and outbound ports.

[0105] After determining the page weights of all web pages based on their update times and the number of inbound and outbound ports, proceed to step 402.

[0106] Step 402: Based on the page layout information and URL call information of all the web pages, score all the web pages to obtain the web page score value of all the web pages.

[0107] After obtaining all the web pages visited by the user, each web page can be rated based on its page layout and URL information to obtain a page rating. Specifically, page tags for a specific area of ​​a web page can be extracted from the page layout information and compared with predefined page tags. This comparison, combined with the URL information of each web page, determines the activity level of each page. In this way, all web pages are rated. In the actual implementation, if the page tags for a specific area of ​​a page match the predefined tags and the page activity is high, the page receives a higher rating; conversely, if the page activity is low, the page receives a lower rating.

[0108] After scoring all web pages based on their layout and URL information to obtain a web page score, proceed to step 403.

[0109] Step 403: Calculate the webpage weight score for all webpages based on the webpage weight and the webpage rating.

[0110] After obtaining the page rating for each page in the entire webpage ecosystem, the page weight score for each page can be calculated based on its corresponding page weight and page rating. Specifically, the product of each page's page rating and page weight can be calculated, and this product can be used as the page weight score for each page.

[0111] After calculating the webpage weight score for all webpages based on their weight and rating, proceed to step 404.

[0112] Step 404: Filter all the web pages according to the web page weight score to obtain the filtered web pages.

[0113] After calculating the webpage weight score for all webpages based on their weight and rating, the webpages can be filtered based on these scores to obtain a filtered list. Specifically, webpages with a weight score greater than a weight score threshold can be identified and selected as the filtered list.

[0114] This application embodiment uses an improved T-HITS algorithm to analyze the dataset and filters all links from web pages to their own websites, effectively filtering out useless web pages and improving the accuracy of web page filtering.

[0115] After filtering all web pages based on their update time and inbound / outbound port information to obtain the filtered web pages, proceed to step 103.

[0116] Step 103: Construct a webpage layout tree based on the webpage pointing relationship corresponding to the filtered webpages.

[0117] After filtering all web pages based on their update time and inbound / outbound port information, a web page layout tree can be constructed based on the corresponding web page linking relationships. Specifically, the web page linking relationships corresponding to the filtered web pages can be determined based on the established web page linking relationship dataset, and a web page layout tree can be constructed based on these relationships.

[0118] The process of building a webpage layout tree can be combined with Figure 5 The following is a detailed description.

[0119] Reference Figure 5 The flowchart illustrates the steps of a webpage layout tree construction method provided in an embodiment of this application, as shown below. Figure 5 As shown, the webpage layout tree construction method may include steps 501 and 502.

[0120] Step 501: Based on the webpage weight scores in descending order, parse the webpage code of the filtered webpages sequentially to obtain the DOM node tree corresponding to each webpage in the filtered webpages.

[0121] In this embodiment, after obtaining the webpage weight score of the filtered webpages, the webpage code of the filtered webpages can be parsed sequentially according to the webpage weight score from largest to smallest, thereby obtaining the DOM node tree corresponding to each webpage in the filtered webpages.

[0122] In this example, the filtered web pages can be rendered by simulating the browser's rendering process to obtain the position information of the element corresponding to each DOM node in the filtered web pages in the browser. In this way, the DOM node tree corresponding to each web page in the filtered web pages can be obtained.

[0123] After obtaining the DOM node tree corresponding to each webpage in the filtered webpages, proceed to step 502.

[0124] Step 502: Construct the webpage layout tree corresponding to the filtered webpage based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimensions.

[0125] After obtaining the DOM node tree corresponding to each webpage in the filtered webpages, a webpage layout tree corresponding to the filtered webpages can be constructed based on the webpage linking relationships, the DOM node tree, and the preset webpage splitting dimensions. The preset webpage splitting dimensions refer to the pre-set webpage splitting dimensions. In this example, to avoid splitting the webpage dimensions too finely, a manually intervened preset webpage splitting dimension can be added. DOM nodes that do not meet the conditions in the DOM node tree are deleted, and the webpage layout tree is constructed based on the webpage linking relationships of the filtered webpages and the remaining DOM nodes of each webpage in the filtered webpages.

[0126] The implementation process of constructing a webpage layout tree by combining webpage pointer relationships, DOM node trees, and pre-defined webpage splitting dimensions can be combined with... Figure 6 The following is a detailed description.

[0127] Reference Figure 6 This illustrates a flowchart of another webpage layout tree construction method provided in an embodiment of this application, such as... Figure 6 As shown, the webpage layout tree construction method may include steps 601, 602 and 603.

[0128] Step 601: Construct an initial webpage layout tree with the filtered webpages as parent nodes according to the webpage pointing relationship.

[0129] In this embodiment, after obtaining the webpage pointing relationship of the filtered webpages, an initial webpage layout tree with the filtered webpages as parent nodes can be constructed based on the webpage pointing relationship.

[0130] The initial webpage layout tree is constructed with the filtered webpages as parent nodes.

[0131] After constructing the initial webpage layout tree with the filtered webpages as parent nodes according to the webpage pointing relationship, proceed to step 602.

[0132] Step 602: Extract the target webpage feature information within the filtered webpage based on the DOM node tree and the preset webpage splitting dimensions.

[0133] After constructing the initial webpage layout tree with the filtered webpages as parent nodes, the DOM nodes of each webpage in the filtered webpages can be filtered according to the preset webpage splitting dimensions, and then the target webpage feature information corresponding to the filtered DOM nodes of each webpage can be extracted.

[0134] After extracting the target webpage feature information within the filtered webpage, proceed to step 603.

[0135] Step 603: Based on the target webpage feature information, construct the child nodes of the parent node to obtain the webpage layout tree.

[0136] After extracting the target webpage feature information from the filtered webpages, the child nodes of the parent node can be constructed based on the target webpage feature information to obtain the webpage layout tree.

[0137] This application embodiment obtains important information about a webpage by simulating the HTML rendering process, which can provide authoritative key information for website construction.

[0138] After constructing the webpage layout tree, proceed to step 104.

[0139] Step 104: Extract the target webpage information of the target webpage tag from the webpage layout tree.

[0140] After constructing the webpage layout tree, the target webpage information corresponding to the target webpage tag can be extracted from the webpage layout tree. Specifically, the target webpage tag of the webpage information to be extracted can be obtained in advance, and the parent and child nodes of the webpage layout tree can be queried in sequence with the target webpage tag to extract the target webpage information corresponding to the target webpage tag.

[0141] The webpage information extraction method provided in this application involves acquiring all webpages accessed by the user, filtering them based on their update time and inbound / outbound port information to obtain filtered webpages, constructing a webpage layout tree based on the corresponding webpage relationships, and extracting target webpage information (target webpage tags) from the layout tree. This application's method effectively filters useless webpages by combining page update time and inbound / outbound port information, thereby improving the accuracy of webpage filtering. Furthermore, the webpage layout tree effectively extracts important information from the webpages.

[0142] Reference Figure 7 The diagram shows a schematic representation of a web page information extraction device according to an embodiment of this application. Figure 7 As shown, the web page information extraction device 700 may include the following modules:

[0143] The All Webpage Acquisition Module 710 is used to acquire all webpages accessed by the user.

[0144] The webpage filtering module 720 is used to filter all webpages based on their page update time and page inbound / outbound port information to obtain filtered webpages.

[0145] The webpage layout tree construction module 730 is used to construct a webpage layout tree based on the webpage pointing relationship corresponding to the filtered webpages.

[0146] The target webpage information extraction module 740 is used to extract target webpage information of target webpage tags from the webpage layout tree.

[0147] Optionally, the all webpage acquisition module 710 includes:

[0148] The URL set crawling unit is used to crawl the set of URLs accessed by the user based on web crawling technology;

[0149] All webpage acquisition units are used to determine all webpages accessed by the user based on the URLs in the URL set.

[0150] Optionally, the device further includes:

[0151] The webpage pointing relationship determination module is used to determine the webpage pointing relationship between all webpages based on the redirection information of the URLs in the URL set;

[0152] The pointer dataset construction module is used to construct a webpage pointer relationship dataset based on all the webpages and the webpage pointer relationships.

[0153] Optionally, the webpage filtering module 720 includes:

[0154] The webpage weight determination unit is used to call the T-HITS algorithm to determine the webpage weight of each webpage based on the page update time and the number of inbound and outbound ports of each webpage.

[0155] The webpage rating unit is used to rate all webpages based on their page layout information and URL call information, and obtain the webpage rating values ​​of all webpages.

[0156] The webpage weight score calculation unit is used to calculate the webpage weight score of all webpages based on the webpage weight and the webpage score.

[0157] The webpage filtering unit is used to filter all webpages according to the webpage weight score to obtain filtered webpages.

[0158] Optionally, the webpage filtering and acquisition unit includes:

[0159] The webpage filtering acquisition subunit is used to acquire webpages whose webpage weight scores are greater than a weight score threshold among all webpages, and to use the webpages whose webpage weight scores are greater than the weight score threshold as the filtered webpages.

[0160] Optionally, the webpage layout tree construction module 730 includes:

[0161] The DOM node tree acquisition unit is used to parse the web page code of the filtered web pages in descending order of the web page weight scores to obtain the DOM node tree corresponding to each web page in the filtered web pages.

[0162] The webpage layout tree construction unit is used to construct the webpage layout tree corresponding to the filtered webpage based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimension.

[0163] Optionally, the webpage layout tree construction unit includes:

[0164] The initial layout tree construction sub-unit is used to construct an initial webpage layout tree with the filtered webpages as parent nodes according to the webpage pointing relationship;

[0165] The target webpage feature extraction subunit is used to extract target webpage feature information within the filtered webpage based on the DOM node tree and the preset webpage splitting dimensions.

[0166] The webpage layout tree construction subunit is used to construct the child nodes of the parent node based on the target webpage feature information to obtain the webpage layout tree.

[0167] The webpage information extraction device provided in this application obtains all webpages accessed by the user, filters all webpages based on their page update time and inbound / outbound port information, obtains filtered webpages, constructs a webpage layout tree based on the webpage pointing relationships corresponding to the filtered webpages, and extracts target webpage information of target webpage tags from the webpage layout tree. This application embodiment effectively filters useless webpages by combining page update time and inbound / outbound port information, thereby improving the accuracy of webpage filtering. Furthermore, by constructing a webpage layout tree, it can effectively extract important information within the webpages.

[0168] This application also provides an electronic device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the above-described web page information extraction method.

[0169] Figure 8 A schematic diagram of the structure of an electronic device 800 according to an embodiment of the present invention is shown. Figure 8As shown, the electronic device 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of the electronic device 800. The CPU 801, ROM 802, and RAM 803 are interconnected via bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0170] Multiple components in electronic device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, microphone, etc.; output unit 807, such as various types of displays, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows electronic device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0171] The various processes and handling described above can be executed by processing unit 801. For example, the methods of any of the above embodiments can be implemented as computer software programs tangibly contained in a computer-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU 801, one or more actions of the methods described above can be performed.

[0172] Additionally, embodiments of this application also provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned web page information extraction method.

[0173] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0174] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, embodiments of this application can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of this application can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0175] This application describes embodiments with reference to flowchart illustrations and / or block diagrams of methods, terminals (systems), and computer program products according to embodiments of this application. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal, causing a series of operational steps to be executed on the computer or other programmable terminal to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] Although preferred embodiments of the present application have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present application.

[0179] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal that includes said element.

[0180] The foregoing has provided a detailed description of a web page information extraction method, a web page information extraction device, an electronic device, and a computer-readable storage medium provided in this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the method and its core ideas. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for extracting information from a webpage, characterized in that, include: Get all the web pages visited by the user; The web pages are filtered based on their update time and inbound / outbound port information to obtain the filtered web pages. Based on the webpage pointing relationships corresponding to the filtered webpages, construct a webpage layout tree; Extract target webpage information from the target webpage tags in the webpage layout tree; The step of filtering all web pages based on their update time and inbound / outbound port information to obtain filtered web pages includes: The T-HITS algorithm is used to determine the page weight of each webpage based on its page update time and the number of inbound and outbound ports. The webpages are then scored based on their page layout and URL call information to obtain a page score value. A page weight score is calculated based on the page weight and the page score value. Finally, the webpages are filtered based on their page weight scores to obtain filtered webpages. The step of constructing a webpage layout tree based on the webpage pointing relationships corresponding to the filtered webpages includes: Based on the webpage weight scores in descending order, the webpage code of the filtered webpages is parsed sequentially to obtain the DOM node tree corresponding to each webpage in the filtered webpages; Based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimensions, construct the webpage layout tree corresponding to the filtered webpage.

2. The method according to claim 1, characterized in that, The process of obtaining all web pages accessed by the user includes: The set of URLs accessed by the user is crawled using web crawling technology; Based on the URLs in the URL set, determine all the web pages accessed by the user.

3. The method according to claim 2, characterized in that, After determining all the web pages visited by the user based on the URLs in the URL set, the method further includes: Based on the redirection information of the URLs in the URL set, determine the webpage pointing relationship between all the webpages; Based on all the web pages and the web page pointing relationships, construct a web page pointing relationship dataset.

4. The method according to claim 1, characterized in that, The step of filtering all web pages based on the web page weight score to obtain filtered web pages includes: The web pages whose weight scores are greater than a weight score threshold are obtained from all the web pages, and these web pages with weight scores greater than the weight score threshold are used as the filtered web pages.

5. The method according to claim 1, characterized in that, The step of constructing the webpage layout tree corresponding to the filtered webpages based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimensions includes: Based on the webpage pointing relationship, construct an initial webpage layout tree with the filtered webpages as parent nodes; Based on the DOM node tree and the preset webpage splitting dimensions, extract the target webpage feature information within the filtered webpage; Based on the target webpage feature information, construct the child nodes of the parent node to obtain the webpage layout tree.

6. A webpage information extraction device, characterized in that, include: The "All Webpages Retrieval" module is used to retrieve all webpages accessed by the user. The webpage filtering module is used to filter all webpages based on their update time and inbound / outbound port information to obtain filtered webpages. The webpage layout tree construction module is used to construct a webpage layout tree based on the webpage pointing relationship corresponding to the filtered webpages; The target webpage information extraction module is used to extract target webpage information of target webpage tags from the webpage layout tree; The filtered webpage acquisition module includes: The webpage weight determination unit is used to call the T-HITS algorithm to determine the webpage weight of each webpage based on the page update time and the number of inbound and outbound ports of each webpage. The webpage rating unit is used to rate all webpages based on their page layout information and URL call information, and obtain the webpage rating values ​​of all webpages. The webpage weight score calculation unit is used to calculate the webpage weight score of all webpages based on the webpage weight and the webpage score. The webpage filtering unit is used to filter all webpages according to the webpage weight score to obtain the filtered webpages. The webpage layout tree construction module includes: The DOM node tree acquisition unit is used to parse the web page code of the filtered web pages in descending order of the web page weight scores to obtain the DOM node tree corresponding to each web page in the filtered web pages. The webpage layout tree construction unit is used to construct the webpage layout tree corresponding to the filtered webpage based on the webpage pointing relationship, the DOM node tree, and the preset webpage splitting dimension.

7. An electronic device, characterized in that, include: A processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the processor, when executing the program, implements the web page information extraction method according to any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, When the instructions in the storage medium are executed by the processor of the electronic device, the electronic device is able to perform the web page information extraction method according to any one of claims 1 to 5.

Images