Website collection method based on xpath configuration automatic analysis

Abstract

The invention discloses a website acquisition method based on xpath configuration automatic analysis, which comprises the following steps of: configuring an xpath rule of a website column list in advance, and if a plurality of column rules exist in a website, not binding the column rules to corresponding columns, and automatically and accurately matching the required column list rule in the plurality of rules, thereby improving the accuracy of the column list. The problems that a large number of list item xpath rules are configured, the rule is required to specify a tedious process corresponding to a target column, and large-scale use cannot be achieved are solved. According to the method, the data collection efficiency is greatly improved, meanwhile, the universality is high, and effective data collection can be carried out as long as column list webpages conform to the SEO standard.

Description

Technical Field

[0001] The present invention relates to the field of website data collection, and in particular to a website collection method based on automatic parsing of XPath configuration. Background Art

[0002] In the current information age, with the rapid development of the Internet, the amount of information on the network has exploded. This data is of great value for research, analysis and decision-making. In order to effectively obtain this information, various network data collection technologies have emerged.

[0003] However, manual collection of website data is inefficient and prone to errors, and cannot cope with large-scale data collection. The traditional full-scale rule configuration method is inefficient and requires the configuration personnel to have certain technical reserves. The configuration process is complicated and time-consuming. Therefore, it is very necessary to develop an automated, efficient, and large-scale website collection method. Summary of the invention

[0004] The purpose of the present invention is to provide a website collection method based on automatic parsing of xpath configuration to solve the problems raised in the above background technology.

[0005] To achieve the above object, the present invention provides the following technical solution: a website collection method based on automatic parsing of xpath configuration, comprising the steps of:

[0006] S1: Configure column xpath rules, and configure corresponding column list recognition rules according to the website structure to be collected, so as to locate the target data in the column webpage;

[0007] S2: Download the source code, use the CDP protocol or webdriver to initiate a request to the column URL of the target website, and obtain the rendered web page source code data after the request is successful;

[0008] S3: rule matching, traverse the xpath list rules configured under the website according to the previously configured rules, and automatically match them with the list area path of the current column page source code until a matching list rule is found;

[0009] S4: Parse the data. Based on the matched list rules in step S3, parse each published data row in the list, and then use DOM or regular expression to extract the a tag in the current row. The href attribute in the a tag is the article body URL address (note that the absurl address must be obtained), and the title or text of the a tag is the article body title; the rightmost time in the data row is the release time of the current article. At this time, the article title, release time, and body address information have all been parsed and obtained. No task rule parameter configuration is required in this process, and all are automatically extracted by this method;

[0010] S5: Data range, based on the list release time parsed in step S4, control the range of data collection. If the release time of the last data of the current page data is greater than or equal to the collection target time, the page turning operation is performed. If the release time of the last data of the current page data is less than the collection target time, the list traversal parsing is terminated. The page turning behavior of the list is that the system simulates the page turning by clicking according to the built-in preset page turning rules of the general CMS in the market. There is no need to manually configure the rules. Special columns can be manually intervened and configured;

[0011] S6: Extract the text, parse the article title, release time, and link information according to steps S1-S5, download the article page source code, and extract the text content and other required attributes from the article page source code;

[0012] S7: Data storage: the extracted data is stored in the database according to preset requirements.

[0013] Preferably, in step S2, the website's column information is traversed using a BFS breadth-first search algorithm throughout the website, and during the traversal, the website's URL address information is analyzed to determine whether the address is a column address. The column address has obvious characteristic information, such as the presence of column identification in the header information, and the content information mostly conforms to the list type. After the BFS traversal of the entire site is completed, the website's column tree is obtained, laying the foundation for subsequent column data collection based on xpath rules.

[0014] Preferably, in step S7, a Bloom filter is introduced to perform filtering during the data storage process.

[0015] Preferably, after the data collected in step S7 is stored in the database, the information of the article is stored in the Bloom filter, and before the next storage, the Bloom filter is used to determine whether the data already exists.

[0016] Compared with the prior art, the present invention has the following beneficial effects:

[0017] The present invention pre-configures an xpath rule for the website column list, and if the website has multiple column rules, it does not need to be bound to the corresponding column. The method can automatically and accurately match the required column list rule among the multiple rules, avoiding the cumbersome process of configuring a large number of list item xpath rules and requiring the rules to specify the corresponding target column and the problem of being unable to be used on a large scale. At the same time, the method can greatly improve the efficiency of data collection because it can automatically match the rule parsing list. In addition, the method has strong versatility, and effective data collection can be performed on any column list webpage that meets the SEO standard. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 is a flow chart of the method of the present invention;

[0019] Figure 2 Schematic diagram of the execution process of the BFS algorithm in the embodiment of the present invention;

[0020] Figure 3 Schematic diagram of the algorithm of the Bloom filter in an embodiment of the present invention. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] See also Figure 1 The present invention provides a technical solution: a website acquisition method based on automatic parsing of xpath configuration, comprising the steps of:

[0023] S1: Configure column xpath rules. According to the website structure to be collected, configure the corresponding column list recognition rules to locate the target data in the column webpage.

[0024] S2: Download the source code, use the CDP protocol or webdriver to initiate a request to the column URL of the target website, and obtain the rendered web page source code data after the request is successful; the column information of the website can generally be obtained through manual configuration and website map; in this embodiment, BFS (breadth-first search algorithm) is used to perform a full-site breadth traversal of the website. During the traversal process, the URL address information of the website is analyzed to determine whether the address is a column address. The column address has obvious characteristic information, such as the presence of column identification in the header information and the content information that mostly conforms to the list type. After the BFS traversal of the entire site is completed, the column tree of the website is obtained, laying the foundation for subsequent column data collection based on xpath rules.

[0025] BFS is the abbreviation of Breadth First Search, which is a graph traversal algorithm. It starts from a given starting node and expands outward layer by layer, first visiting the adjacent nodes of the starting node, then visiting the adjacent nodes of the adjacent nodes, and so on, until all reachable nodes are traversed. The specific BFS algorithm process is as follows:

[0026] (1) Create a queue and add the starting node to the queue;

[0027] (2) Mark the starting node as visited;

[0028] (3) When the queue is not empty, perform the following operations in a loop: a. dequeue a node; b. visit the node; c. queue all unvisited adjacent nodes of the node and mark them as visited;

[0029] (4) The traversal is completed.

[0030] The characteristic of the BFS algorithm is that it traverses the hierarchy, first visiting the node closest to the starting node, then visiting the node one level further away from the starting node, and so on. Since BFS needs to use a queue to assist in its implementation, it can ensure that each node is visited and only visited once.

[0031] The BFS algorithm is widely used in graph traversal, shortest path, connectivity detection and other problems. Figure 2 The execution process of the BFS algorithm is: 1->2,3,4,5->6,7,8,9,10->11,12,13->14;

[0032] S3: Rule matching. According to the previously configured rules, this method will traverse the xpath list rules configured under the website and automatically match them with the list area path of the current column page source code until a matching list rule is found.

[0033] S4: Parse the data. Based on the automatic matching of the list rules in the previous step, you can parse each row of published data in the list, and then use DOM or regular expressions to extract the a tag in the current row. The href attribute in the a tag is the URL address of the article body (note that the absurl address must be obtained), and the title or text of the a tag is the title of the article body. The rightmost time in the data row is the publishing time of the current article. At this time, the title, publishing time, and body address information of the article have all been parsed and obtained. No task rule parameter configuration is required in this process, and all are automatically extracted by this method.

[0034] S5: Data range: Based on the list release time parsed in the previous step, the data collection range can be controlled. If the release time of the last data on the current page is greater than or equal to the collection target time, the page turning operation is performed. If the release time of the last data on the current page is less than the collection target time, the list traversal parsing is terminated. The page turning behavior of the list is simulated by the system according to the built-in preset page turning rules of the general CMS in the market. There is no need to manually configure the rules. Special columns can be manually intervened and configured.

[0035] S6: Text extraction: The previous steps have successfully parsed the information of the article title, release time, and article link. This step mainly downloads the page source code of the article based on the article link, and extracts the text content and other required attributes from the article page source code.

[0036] S7: Data storage: At this point, the collection process of a news list column is completed, and you only need to clean the data and store it in the database as required. There is a problem when storing data in the database: duplicate data does not need to be stored in the database, but the method of using database unique restrictions or pre-storage queries is extremely performance-consuming. The embodiment of the present invention introduces a Bloom filter for processing. After the collected data is stored in the database, the information of the article is stored in the Bloom filter. Before the next data is stored in the database, the Bloom filter is used to determine whether the data already exists.

[0037] Bloom filter is a probabilistic data structure used to determine whether an element is in a set. It features efficient insertion and query, and can determine whether a string must exist or may exist. Bloom filter does not store specific data, so it takes up little space. There are errors in the query results, but the errors are controllable, and it does not support deletion operations.

[0038] When an element is added to the bitmap, k hash functions are used to map the element to k points of the bitmap and set them to 1. When searching, k hash functions are used to check whether all k points of the bitmap are 1. If there are points that are not 1, the key is considered not to exist. If all are 1, it may exist. Bloom filters do not support deletion operations because each slot in the bitmap has only two states (0 or 1). A slot is set to 1, but it is not certain how many times it has been set. It is also not known how many key hashes it has been mapped to, and which hash function it has been mapped to. It is worth noting that as long as one slot is 0, the key must not exist. If all slots mapped by the key are 1, it cannot be said that it must exist, but only that it may exist (false positive rate). The principle of the algorithm is shown in Figure 3:

[0039] In order to better illustrate the implementation mode of the present invention, a specific website case (the "Company News" column of the official website of Shanghai XXX Co., Ltd.) is used as an example for explanation below:

[0040] The xpath rules for the list column of this website include / / div[@class='product-cont'] and / / div[@class='d_list_wrap'] / ul / li. After downloading the source code of the company dynamics column, the two rules configured above are traversed and matched with the paths in the source code of the company dynamics column one by one. When the rule / / div[@class='product-cont'] of the product center is matched, it does not match, and then / / div[@class='d_list_wrap'] / ul / li will be matched, and it is found to be a matching rule.

[0041] An example of DOM content for each line of company news:

[0042] <li class＝"dt_l_li">

[0043] <div class＝"dt_l_info">

[0044] <div class＝"l_txt">

[0045] <div class＝"l_txt_left">

[0046] <p class＝"l_date">2024-05-27

[0047]

[0048] <div class＝"l_txt_r">

[0049] <a href＝"https: mp.weixin.qq.com s xktfdbwxlkwjymzbyesj2q"><h4class="l_title"> XXXX Company won the bid for the China Meteorological Administration's government website and government new media monitoring service project

[0050] <p class＝"l_date1">2024-05-27

[0051] <p class＝"l_summary">

[0052]

[0053]

[0054]

[0055] This method will extract the a tag element from the DOM of each row: <a href＝"https: mp.weixin.qq.com s xktfdbwxlkwjymzbyesj2q"><h4class="l_title"> XXX Company won the bid for the China Meteorological Administration's government website and government new media monitoring service project , href content of a tag;

[0056] https: / / mp.weixin.qq.com / s / xKTfDBwxLKwJymZByeSj2Q is the article link, and the text in the a tag "XXX company won the bid for the China Meteorological Administration's government website and government new media monitoring service project" is the title of the article.<pclass="l_date"> 2024-05-27 You can extract the time when the article was published.

[0057] <div class＝"xwzx-fenye clear pagination">

[0058] <div class＝"xwzx-fenye-list">

[0059] <! --Home-->

[0060] <span class＝"page-item active"> <a href＝"javascript:void(0)；"class＝"page-link"pageindex＝"0"> front page

[0061] <!--Previous Page-->

[0062] <span class＝"page-item disabled"> <a href＝"javascript:void(0)；"class＝"page-link"aria-label＝"previous"pageindex＝"-1"> <span aria-hidden＝"true"> Previous page

[0063] <!--Current page number-->

[0064] <span class＝"cur_page">1

[0065] <!--Next Page-->

[0066] <span class＝"page-item"> <a href＝"javascript:void(0)；"class＝"page-link"aria-label＝"next"pageindex＝"1"> <span aria-hidden＝"true"> Next Page

[0067] <!--Last Page-->

[0068] <span class＝"page-item"> <a href＝"javascript:void(0)；"class＝"page-link"pageindex＝"2"> Last page

[0069] <a class＝"xwzx-fenye-jilu">A total of 27 3 Page

[0070]

[0071]

[0072] The page turning element DOM of this column is as above, and its page turning is <a href＝"javascript:void(0)；"class＝"page-link"aria-label＝"next"pageindex＝"1"> <span aria-hidden＝"true"> Next Page It will automatically match the system's built-in page turning rule / / a[contains(text(),'next page')]. By simulating clicking on this element, you can achieve the page turning effect and obtain the data on the subsequent pages.

[0073] Through the implementation of the above cases, we can realize a website collection method based on very few xpath rule configurations for automatic parsing of column lists, which improves the efficiency of website data collection, enhances adaptability, reduces complexity, and greatly reduces the impact of website structure changes on the collection process.

[0074] The above embodiments can be implemented in whole or in part by software, hardware, firmware or any other combination. When implemented by software, the above embodiments can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions or computer programs. When the computer instructions or computer programs are loaded or executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from one website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that contains one or more available media sets. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium. The semiconductor medium can be a solid-state hard disk.

[0075] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0076] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0077] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only one, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0078] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0079] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0080] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0081] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

[0082] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

[0083] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A website acquisition method based on automatic parsing of xpath configuration, characterized in that: Includes steps: S1: Configure column xpath rules, and configure corresponding column list recognition rules according to the website structure to be collected, so as to locate the target data in the column webpage; S2: Download the source code, use the CDP protocol or webdriver to initiate a request to the column URL of the target website, and obtain the rendered web page source code data after the request is successful; S3: rule matching, traverse the xpath list rules configured under the website according to the previously configured rules, and automatically match them with the list area path of the current column page source code until a matching list rule is found; S4: Parse the data. Based on the matched list rules in step S3, parse each line of published data in the list, and then use DOM or regular expression to extract the a tag in the current line. The href attribute in the a tag is the URL address of the article body, and the title or text of the a tag is the title of the article body. S5: Data range: based on the list release time parsed in step S4, control the range of data collection. If the release time of the last data item on the current page is greater than or equal to the collection target time, turn the page. If the release time of the last data item on the current page is less than the collection target time, end the list traversal parsing. S6: Extract the text, parse the article title, release time, and link information according to steps S1-S5, download the article page source code, and extract the text content and other required attributes from the article page source code; S7: Data storage: the extracted data is stored in the database according to preset requirements.

2. A website acquisition method based on automatic parsing of xpath configuration according to claim 1, characterized in that: In step S2, the website's column information is traversed using the BFS breadth-first search algorithm. During the traversal, the website's URL address information is analyzed to determine whether the address is a column address, so that the column tree information can be quickly obtained, reducing manual configuration.

3. A website acquisition method based on automatic parsing of xpath configuration according to claim 1, characterized in that: The page turning behavior of the list in step S5 is that the system simulates clicking and turning pages according to the built-in preset page turning rules of the common cms on the market, and there is no need to manually configure the rules.

4. A website acquisition method based on automatic parsing of xpath configuration according to claim 1, characterized in that: In the step S7, a Bloom filter is introduced to perform filtering during the data storage process.

5. A website acquisition method based on automatic parsing of xpath configuration according to claim 1, characterized in that: After the data collected in step S7 is stored in the database, the information of the article is stored in the Bloom filter. Before the next data is stored in the database, the Bloom filter is used to determine whether the data already exists.

Images