A data crawling method and device

Abstract

The present disclosure provides a data crawling method and device, relates to the technical field of network application, and can solve the problems of low efficiency and easy errors of the current data crawling method. The specific technical scheme is: a crawler template is configured, and website information to be accessed is obtained; a crawler module is used to set the database structure of a target database; a corresponding target website page is loaded according to the website information to be accessed; a selection operation of a user on the target website page is obtained, and a target position is determined according to the selection operation; a document structure of corresponding content in the target website page is obtained according to the target position, and an XPath expression is generated according to the document structure; a crawler program is called, and target data of the target website page crawled by the crawler program according to the XPath expression is stored in the target database. The present application is used for data crawling.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of network application, and in particular, to a data crawling method and device. BACKGROUND

[0002] A web crawler is a program that automatically extracts web page content, which extracts Internet information according to certain rules. The web crawler generally starts from one or several initial web page URLs (Uniform Resource Locator), obtains the content on the initial web page, and continues to extract new URLs that meet the rules, and puts them into a queue, and repeats the process until it stops when it meets certain conditions set by the system.

[0003] In the prior art, a technical personnel who masters the XPath syntax interprets the page document structure in combination with existing work experience, writes an XPath expression, and a crawler program parses the page using the XPath expression and stores data in a database. However, this method requires a large amount of crawler configuration work, requires the configuration personnel to understand the XPath syntax and html technology, and is prone to errors. SUMMARY

[0004] The embodiments of the present disclosure provide a data crawling method and device, which can solve the problem of low efficiency and error-prone of the current data crawling method. The technical solution is as follows:

[0005] According to a first aspect of the embodiments of the present disclosure, a data crawling method is provided, which comprises:

[0006] A crawler template is configured and website information to be accessed is obtained, the crawler module is used to set the database structure of a target database; a corresponding target website page is loaded according to the website information to be accessed; a selection operation of a user on the target website page is obtained, and a target position is determined according to the selection operation; a document structure of corresponding content in the target website page is obtained according to the target position, and an XPath expression is generated according to the document structure; a crawler program is called, and target data of the target website page crawled by the crawler program according to the XPath expression is stored in the target database.

[0007] The application loads a corresponding website page according to website information to be accessed, obtains a selection operation of a user on the website page to be crawled, determines a target position according to the selection operation, obtains a document structure of corresponding content in the website page according to the target position, generates an XPath expression according to the document structure, finally calls a crawler program, and stores target data of the target website page crawled by the crawler program according to the XPath expression in a target database. Compared with the prior art, the application mainly loads a website to be crawled into a system by using a reverse proxy technology, solves the problem that a browser cannot obtain a non-system document structure due to cross-domain, thereby realizing the analysis of a webpage document in a click and word selection manner and generating an XPath expression. Page data can be obtained quickly, accurately and timely in a what-you-see-is-what-you-get manner, and the correctness of the expression can be tested through an interactive manner.

[0008] In one embodiment, the loading of a corresponding target website page according to the website information to be accessed comprises:

[0009] The website information to be accessed is sent to a proxy server, so that the proxy server sends the website information to be accessed to a target server;

[0010] The target website page sent by the proxy server is received, wherein the target website page is fed back by the target server to the proxy server.

[0011] In one embodiment, the obtaining of a document structure of corresponding content in the target website page according to the target position and the generation of an XPath expression according to the document structure comprise:

[0012] The corresponding content in the target website page is obtained according to the target position;

[0013] The node order of the corresponding content in the target website page is found according to the corresponding content in the target website page by using a recursive method;

[0014] The document structure of the corresponding content in the target website page is obtained according to the node order;

[0015] The XPath expression is generated according to the document structure.

[0016] In one embodiment, after the calling of the crawler program and the storing of target data of the target website page crawled by the crawler program according to the XPath expression in the target database, the method further comprises:

[0017] It is detected whether the target data is consistent with a preset result, and if the target data is not consistent with the preset result, it is detected whether a regular expression is configured.

[0018] If no regular expression is configured, an adjustment operation of a user on the XPath expression is acquired to obtain an adjusted XPath expression;

[0019] Data of a website page to be crawled is crawled by using the adjusted XPath expression until the target data and the preset result are consistent.

[0020] In an embodiment, the method further comprises: detecting whether the target data and the preset result are consistent, and if the target data and the preset result are inconsistent, configuring a regular expression and performing secondary matching on the target data of the target website page crawled by using the XPath expression by using the regular expression.

[0021] According to a second aspect of the embodiments of the present disclosure, a data crawling device is provided, which comprises:

[0022] A configuration module is configured to configure a crawler template and acquire website information to be accessed, and the crawler module is configured to set a database structure of a target database;

[0023] A loading module is configured to load a corresponding target website page according to the website information to be accessed;

[0024] An acquisition module is configured to acquire a selection operation of a user on the target website page and determine a target position according to the selection operation;

[0025] A processing module is configured to acquire a document structure of corresponding content in the website page according to the target position and generate an XPath expression according to the document structure;

[0026] A calling module is configured to call a crawler program and store target data of the target website page crawled by the crawler program according to the XPath expression in the target database.

[0027] In an embodiment, the loading module is further configured to:

[0028] send the website information to be accessed to a proxy server, so that the proxy server sends the website information to be accessed to a target server;

[0029] receive a target website page sent by the proxy server, the target website page being fed back by the target server to the proxy server.

[0030] In an embodiment, the processing module is further configured to:

[0031] acquire corresponding content in the target website page according to the target position;

[0032] Based on the corresponding content in the target website page, a recursive method is used to find the node order of the corresponding content in the website page;

[0033] The document structure of the corresponding content in the target website page is obtained according to the node order;

[0034] Generate an XPath expression based on the document structure.

[0035] In one embodiment, the apparatus further includes a detection module, the detection module being used for:

[0036] Check whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, check whether a regular expression is configured.

[0037] If no regular expression is configured, the user's adjustment operation on the XPath expression is obtained to get the adjusted XPath expression;

[0038] The data from the target website page is crawled using the adjusted XPath expression until the target data matches the preset result.

[0039] In one embodiment, the detection module is further configured to:

[0040] The system checks whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, a regular expression is configured, and the regular expression is used to perform a secondary matching on the target data of the target website page to be crawled by the XPath expression.

[0041] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Attached Figure Description

[0042] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.

[0043] Figure 1 This is a flowchart of a data scraping method provided in an embodiment of this disclosure;

[0044] Figure 2 This is a schematic diagram of a website information to be accessed provided in an embodiment of this disclosure;

[0045] Figure 3 This is a structural diagram of a data capture device provided in an embodiment of this disclosure. Detailed Implementation

[0046] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this disclosure. Rather, they are merely examples of systems and methods consistent with some aspects of this disclosure as detailed in the appended claims.

[0047] This disclosure provides a data scraping method, such as... Figure 1 As shown, the method includes the following steps:

[0048] 101. Configure the crawler template and obtain the website information to be accessed.

[0049] The crawler module is used to set the database structure of the target database. After the database structure is set, the crawler program writes the results into the corresponding database table.

[0050] This involves creating crawler templates based on business needs, obtaining website information to be visited based on offline research, and generating a website list. Specific website information to be visited includes, for example... Figure 2 As shown.

[0051] Specifically, add crawler configurations to each of the surveyed websites and input the website links from the list into the "Website Entry Links" configuration item.

[0052] 102. Load the corresponding target website page based on the website information to be accessed.

[0053] In one embodiment, loading the corresponding website page based on the website information to be accessed includes:

[0054] Send the website information to be accessed to the proxy server, so that the proxy server can send the website information to be accessed to the target server;

[0055] Receive the target website page sent by the proxy server. The target website page is the one that the target server sends back to the proxy server.

[0056] In practical applications, after entering the website links from the website list into the configuration item "Website Entry Links" and clicking "Generate Expression", the system will proxy the original website into a page of its own and display it to the configuration personnel.

[0057] 103. Obtain the user's selection action on the target website page, and determine the target location based on the selection action.

[0058] Specifically, the configuration personnel select page content on the loaded website page according to the predefined crawler template, and the program determines the target location based on the click location.

[0059] 104. Obtain the document structure of the corresponding content in the target website page based on the target location, and generate an XPath expression based on the document structure.

[0060] In one embodiment, the document structure of the corresponding content on the website page is obtained based on the target location, and an XPath expression is generated based on the document structure, including:

[0061] Retrieve the corresponding content from the website page based on the target location. Then, using a recursive method, find the node order of the corresponding content from the website page. Based on the node order, obtain the document structure of the corresponding content from the website page. Finally, generate an XPath expression based on the document structure.

[0062] Specifically, after determining the target location based on the clicked location, the program uses a recursive method to search for the parent node level by level until the root node is found, and generates an XPath expression according to the order of the nodes.

[0063] 105. Call the web crawler and store the target data of the target website pages crawled by the web crawler according to the XPath expression in the target database.

[0064] Specifically, the task scheduling system schedules task executors, which in turn call crawler programs. The crawler programs, based on their configuration and settings, crawl the corresponding website data and convert the original unstructured webpage data into structured data before storing it in the target database.

[0065] After step 105, the method further includes: detecting whether the target data and the preset result are consistent; if the target data and the preset result are inconsistent, detecting whether a regular expression is configured.

[0066] If no regular expression is configured, the user's adjustment operation on the XPath expression is obtained to get the adjusted XPath expression;

[0067] The modified XPath expression is used to crawl the data of the website page to be crawled until the target data matches the preset result.

[0068] In one embodiment, the method further includes: detecting whether the target data and the preset result are consistent; if the target data and the preset result are inconsistent, configuring a regular expression and using the regular expression to perform secondary matching on the target data of the website page to be crawled by the XPath expression.

[0069] In practical applications, clicking the test button allows you to quickly check if the generated XPath expression matches your expectations. If not, you can fine-tune the expression. If the XPath expression fails to achieve the expected results, you can configure regular expressions to perform secondary matching within the parsed content of the XPath expression, thus making the results more accurate.

[0070] After the expression is configured, you can click the "Test" button to directly request the crawler program. The crawler program will then hand over the crawled data to this system, which will display the data to the operator to test the overall configuration results and check whether the interface matches expectations in a timely manner.

[0071] This application configures a crawler template and obtains information about the website to be accessed. Based on this information, it loads the corresponding website page, obtains the user's selection operation on the website page based on the crawler template, determines the target location based on the selection operation, obtains the document structure of the corresponding content on the website page based on the target location, generates an XPath expression based on the document structure, and finally calls the crawler program. The target data of the target website page crawled by the crawler program based on the XPath expression is then stored in a target database. Compared with existing technologies, this application mainly uses reverse proxy technology to load the website to be crawled into its own system, solving the problem that browsers cannot obtain document structures from non-local systems when crossing domains. This allows for parsing web page documents using a click-and-select method to generate XPath expressions. Page data can be obtained quickly, accurately, and promptly in a WYSIWYG manner, and the correctness of the expressions can be tested interactively.

[0072] This disclosure also provides a data capture device, such as... Figure 3 As shown, the device 30 includes: a configuration module 301, used to configure the crawler template and obtain information about the website to be accessed; the crawler module is used to set the database structure of the target database.

[0073] Loading module 302 is used to load the corresponding target website page based on the website information to be accessed;

[0074] The acquisition module 303 is used to acquire the user's selection operation on the target URL page and determine the target location based on the selection operation;

[0075] Processing module 304 is used to obtain the document structure of the corresponding content in the target website page according to the target location, and generate an XPath expression based on the document structure;

[0076] Module 305 is used to invoke the crawler program and store the target data of the target website pages crawled by the crawler program according to the XPath expression in the target database.

[0077] In one embodiment, the loading module 304 is further configured to: send the website information to be accessed to the proxy server, so that the proxy server sends the website information to be accessed to the target server;

[0078] Receive the website pages to be crawled from the proxy server. The website pages to be crawled are the ones that the target server sends back to the proxy server.

[0079] In one embodiment, the processing module 304 is further configured to: obtain the corresponding content in the website page according to the target location; find the node order of the corresponding content in the website page using a recursive method according to the corresponding content in the website page; obtain the document structure of the corresponding content in the website page according to the node order; and generate an XPath expression according to the document structure.

[0080] In one embodiment, the device further includes a detection module 306, which is used for:

[0081] Check if the target data and the preset results are consistent. If the target data and the preset results are inconsistent, check if a regular expression is configured.

[0082] If no regular expression is configured, the user's adjustments to the XPath expression are retrieved to obtain the adjusted XPath expression.

[0083] The modified XPath expression is used to crawl the data of the website page to be crawled until the target data matches the preset result.

[0084] In one embodiment, the detection module 306 is further configured to:

[0085] The system checks whether the target data matches the preset results. If the target data does not match the preset results, it configures regular expressions and uses them to perform secondary matching on the target data of the website page to be crawled by the XPath expression.

[0086] Based on the methods described in the above embodiments, this disclosure also provides a computer-readable storage medium. For example, a non-transitory computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, magnetic tape, a floppy disk, and an optical data storage device, etc. This storage medium stores computer instructions for executing the data capture methods described in the above embodiments, which will not be elaborated further here.

[0087] Other embodiments of this disclosure will readily occur to those skilled in the art upon consideration of the specification and practice of the disclosure herein. This application is intended to cover any variations, uses, or adaptations of this disclosure that follow the general principles of this disclosure and include common knowledge or customary techniques in the art not disclosed herein. The specification and examples are to be considered exemplary only, and the true scope and spirit of this disclosure are indicated by the following claims.

[0088] It should be understood that this disclosure is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this disclosure is limited only by the appended claims.

Claims

1. A data scraping method, characterized in that, The method includes: Configure a crawler template and obtain information about the website to be accessed. The crawler template is used to set the database structure of the target database. Load the corresponding target website page based on the website information to be accessed; Obtain the user's selection operation on the target website page, and determine the target location based on the selection operation; Obtain the document structure of the corresponding content in the target website page based on the target location, and generate an XPath expression based on the document structure; The web crawler is invoked, and the target data of the target website page crawled by the web crawler according to the XPath expression is stored in the target database.

2. The method according to claim 1, characterized in that, The step of loading the corresponding target website page based on the website information to be accessed includes: The website information to be accessed is sent to the proxy server, so that the proxy server can send the website information to be accessed to the target server. Receive the target website page sent by the proxy server, wherein the target website page is fed back to the proxy server by the target server.

3. The method according to claim 1, characterized in that, The step of obtaining the document structure of the corresponding content in the target website page based on the target location, and generating an XPath expression based on the document structure, includes: Obtain the corresponding content from the target website page based on the target location; Based on the corresponding content in the target website page, a recursive method is used to find the node order of the corresponding content in the target website page; The document structure of the corresponding content in the target website page is obtained according to the node order; Generate an XPath expression based on the document structure.

4. The method according to claim 1, characterized in that, After invoking the web crawler and storing the target data of the target website page crawled by the web crawler according to the XPath expression in the target database, the method further includes: Check whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, check whether a regular expression is configured. If no regular expression is configured, the user's adjustment operation on the XPath expression is obtained to get the adjusted XPath expression; The adjusted XPath expression is used to crawl the data of the target website page until the target data matches the preset result.

5. The method according to claim 4, characterized in that, The method further includes: The system checks whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, a regular expression is configured, and the regular expression is used to perform a secondary matching on the target data of the target website page crawled by the XPath expression.

6. A data acquisition device, characterized in that, The device includes: The configuration module is used to configure the crawler template and obtain the website information to be accessed. The crawler template is used to set the database structure of the target database. The loading module is used to load the corresponding target website page based on the website information to be accessed. The acquisition module is used to acquire the user's selection operation on the target website page and determine the target location based on the selection operation; The processing module is used to obtain the document structure of the corresponding content in the target website page according to the target location, and generate an XPath expression according to the document structure; The calling module is used to call the crawler program and store the target data of the target website page crawled by the crawler program according to the XPath expression in the target database.

7. The apparatus according to claim 6, characterized in that, The loading module is also used for: The website information to be accessed is sent to the proxy server, so that the proxy server can send the website information to be accessed to the target server. Receive the target website page sent by the proxy server, wherein the target website page is fed back to the proxy server by the target server.

8. The apparatus according to claim 6, characterized in that, The processing module is also used for: Obtain the corresponding content from the target website page based on the target location; Based on the corresponding content in the target website page, a recursive method is used to find the node order of the corresponding content in the target website page; The document structure of the corresponding content in the target website page is obtained according to the node order; Generate an XPath expression based on the document structure.

9. The apparatus according to claim 6, characterized in that, The device further includes a detection module, the detection module being used for: Check whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, check whether a regular expression is configured. If no regular expression is configured, the user's adjustment operation on the XPath expression is obtained to get the adjusted XPath expression; The data from the target website page is crawled using the adjusted XPath expression until the target data matches the preset result.

10. The apparatus according to claim 9, characterized in that, The detection module is also used for: The system checks whether the target data and the preset result are consistent. If the target data and the preset result are inconsistent, a regular expression is configured, and the regular expression is used to perform a secondary matching on the target data of the target website page crawled by the XPath expression.

Images