Data crawling method, device, storage medium and system

Abstract

The invention is suitable for the technical field of network information, and provides a data crawling method and device, a storage medium and a system. The method comprises the following steps: acquiring a webpage screenshot of a to-be-analyzed page to obtain a plurality of labeled positions; acquiring a data interface at the back end of the webpage corresponding to the labeling position; obtaining response data from the data stream read from the data interface; obtaining a data structure of the response data, and obtaining an injection data string; obtaining an injection page; and performing data crawling on the to-be-crawled information based on the positioning position. According to the scheme, only the position of the to-be-crawled information of the page appearing in the page needs to be labeled, the position of the data at the labeled position in the back-end data of the page can be quickly obtained, and automatic crawling of the target data can be achieved based on the position. A large amount of manual analysis and matching function construction for each change of page background data are not needed, so that the cost and difficulty of manual matching are greatly saved.

Description

Technical Field

[0001] The present invention belongs to the field of network information technology, and in particular relates to a data crawling method, device, storage medium and system. Background Art

[0002] Data crawling refers to the process of automatically extracting information from web pages or other data sources on the Internet by writing programs. It is usually part of a web crawler, which is used to crawl, index and analyze web page content. For example, a terminal periodically crawls weather information from a server to update the local weather data display.

[0003] Existing data crawling technology usually relies on field name matching to locate the target field and obtain the required crawled data. Specifically, it extracts the required data by identifying elements with specific structures and identifiers (such as HTML tags, class names, IDs or field names, etc.) in the background data of the web page. This method mainly locates and extracts information through static field identifiers (such as attributes of HTML tags). And because data crawling relies more on the English naming of field names, some field names may be unclear or change frequently in practice, which will lead to inaccurate positioning of the target field or the need for synchronous and regular maintenance to keep the match available.

[0004] Therefore, existing data capture methods often rely too much on direct identification and matching of fields, making it difficult to ensure the consistency and validity of the captured data. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a data crawling method, which aims to solve the problem that the existing data crawling methods often rely too much on direct identification and matching of fields, resulting in difficulty in ensuring the consistency and validity of the crawled data.

[0006] The embodiment of the present application is implemented by providing a data crawling method, the method comprising: Obtain a webpage screenshot of the page to be analyzed, and mark the information to be crawled on the webpage screenshot to obtain a number of marked positions; Obtaining the data interface of the webpage backend corresponding to the marked position; Calling the data interface to obtain response data from the data stream read from the data interface; Analyze the response data to obtain a data structure of the response data, and use the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers; Sending a data request from the page end to the original target address, intercepting a response data string, injecting the simulated response into a corresponding position in the response data string, and obtaining an injected data string; the page end performs loading based on the injected data string to obtain an injected page; The unique identifier in the injected page is retrieved, and then the location of the data field corresponding to the information to be crawled in the back end of the web page is obtained, and data crawling is performed on the information to be crawled based on the location.

[0007] Another object of an embodiment of the present application is to provide a data crawling device, the data crawling device comprising: The marked position acquisition module is used to obtain a webpage screenshot of the page to be analyzed, mark the information to be crawled on the webpage screenshot, and obtain a number of marked positions; A data interface acquisition module, used to acquire the data interface of the web page backend corresponding to the marked position; A response data acquisition module, used for calling the data interface and obtaining response data from the data stream read from the data interface; A simulation response acquisition module, used for analyzing the response data, obtaining a data structure of the response data, and using the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers; The injection page acquisition module is used to send an initial data request from the page end to the original target address, intercept the response data string, inject the simulated response into the corresponding position in the response data string, and obtain the injection data string; the page end is loaded based on the injection data string to obtain the injection page; The data crawling module is used to retrieve the unique identifier in the injected page, and then obtain the location of the data field corresponding to the information to be crawled in the back end of the web page, and crawl the information to be crawled based on the location.

[0008] Another object of an embodiment of the present application is to provide a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the data crawling method as described above.

[0009] Another object of an embodiment of the present application is to provide a data crawling system, comprising a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the data crawling method as described above.

[0010] A data crawling method provided by the embodiment of the present application has the outstanding advantage that: based on this scheme, it is only necessary to mark the position of the information to be crawled on the page in the page, so that the position of the data at the marked position in the back-end data of the page can be quickly obtained, and the target data can be automatically crawled based on the position. The present application replaces the traditional scheme of first performing back-end data analysis and then constructing a matching formula for data acquisition with position annotation, and there is no need to perform a large amount of manual analysis and construct matching functions for each change of the back-end data of the page, which greatly saves the cost and difficulty of manual matching. In addition, considering that the layout of the page display does not change frequently, the data format, page architecture or variable name of the back-end information to be analyzed may change frequently. In the traditional scheme, manual data analysis is required for each change, and the data format of the information to be analyzed is obtained and then the matching formula is constructed for matching, and the overall real-time difficulty and implementation cost are obviously higher. The present application automatically locates the position of the information to be crawled by injection for data collection, significantly improves the speed and accuracy of field acquisition and verification, is not affected by the uncertainty of the back-end data return, reduces the dependence on the field naming rules, and improves the accuracy and reliability of data crawling. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An application environment diagram of the data crawling method provided in an embodiment of the present application; Figure 2 A flowchart of a data crawling method provided in an embodiment of the present application; Figure 3 A schematic diagram of unmodified original response data provided in an embodiment of the present application; Figure 4 A schematic diagram of response data including several unique identifiers provided in an embodiment of the present application; Figure 5 A schematic diagram of a data interface for locating a specific location based on location coordinates provided in an embodiment of the present application; Figure 6 A structural block diagram of a data crawling device provided in an embodiment of the present application; Figure 7 FIG. 4 is a block diagram of the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0012] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0013] It is understood that the terms "first", "second", etc. used in this application may be used herein to describe various elements, but unless otherwise specified, these elements are not limited by these terms. These terms are only used to distinguish a first element from another element. For example, a first script may be referred to as a second script, and similarly, a second script may be referred to as a first script without departing from the scope of this application.

[0014] Figure 1 The application environment diagram of the data crawling method provided in the embodiment of the present application is as follows: Figure 1 As shown, in the application environment, a terminal 110 and a computer device 120 are included.

[0015] The computer device 120 may be an independent physical server or terminal, or a server cluster composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud servers, cloud databases, cloud storage, and CDN.

[0016] The terminal 110 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, etc., but is not limited thereto. The terminal 110 and the computer device 120 may be connected via a network, and this application does not limit this.

[0017] like Figure 2 As shown, in one embodiment, a data crawling method is proposed. This embodiment mainly applies this method to the above Figure 1 A data crawling method may include the following steps: Step S10, obtaining a webpage screenshot of the page to be analyzed, marking the information to be crawled on the webpage screenshot, and obtaining a number of marked positions.

[0018] In this embodiment, an automated browser such as Selenium can be used to load a web page and take a screenshot. The browser automation tool can be used to simulate the user's operation and capture an image of the entire web page. Then mark the location of the information to be crawled. If manual marking is required, an image editing tool can be used to manually mark the area of ​​information on the screenshot. If automatic marking is required, computer vision technology, such as a deep learning object detection algorithm, can be used to identify the text or content in a specific area of ​​the web page for marking.

[0019] Step S20, obtaining the data interface of the web page backend corresponding to the marked position.

[0020] In this embodiment, the developer tools of the browser may be used to observe the network requests during the web page loading process, such as Ajax requests to obtain the data interface.

[0021] Step S30, calling the data interface, and obtaining response data from the data stream read from the data interface.

[0022] In this embodiment, a script may be written through an API testing tool or using Python's requests library to call a data interface to obtain normal response data of the original web page request without modification.

[0023] Step S40, analyzing the response data to obtain a data structure of the response data, using the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers.

[0024] In this embodiment, by viewing the returned JSON or XML data, analyzing the hierarchical structure and each field of the data, generating a schema template, and finally reversely constructing the data according to the schema template so that each data has unique identification. Among them, the response data may contain lists, dictionaries, or nested structures. According to the structure obtained by analysis, a simulated response containing necessary fields is constructed, especially the unique identifier field needs to be inserted. For example, a character table formed by specific symbols or content: "item_id": "def55555", "name": "Product66666".

[0025] Step S50, sending a data request from the page end to the original target address, intercepting the response data string, injecting the simulated response into the corresponding position in the response data string, and obtaining the injected data string; the page end loads based on the injected data string to obtain the injected page.

[0026] In this embodiment, a browser plug-in or the like can be used to intercept and modify the request and response data sent to the backend. In the intercepted response data, the previously constructed simulated response is inserted into the correct position. For example, if there is an items field in the response data, the item_list in the simulated data replaces the corresponding part in the original data.

[0027] S60, retrieving the unique identifier in the injected page, and then obtaining the location of the data field corresponding to the information to be crawled in the back end of the web page, and crawling the information to be crawled based on the location.

[0028] In this embodiment, the page end will re-render the page according to the injected data, and there is no need to intervene in the page loading process, because only the data needs to be intercepted and modified, and the page will automatically render based on the injected data. In the injected page, the actual location of the data field is determined based on the unique identifier, and then the data is crawled. Through retrieval and comparison, users can quickly locate the corresponding relationship of the data fields.

[0029] In the embodiment of the present application, it is only necessary to mark the position of the information to be crawled on the page where it appears in the page, so that the position of the data at the marked position in the backend data of the page can be quickly obtained, and the target data can be automatically crawled based on the position. The present application replaces the traditional solution of first performing backend data analysis and then constructing a matching formula for data acquisition with position annotation, and there is no need to perform a large amount of manual analysis and construct matching functions for each change in the backend data of the page, which greatly saves the cost and difficulty of manual matching.

[0030] In addition, since the page display format does not change frequently, but the backend data format, page structure, or variable name of the information to be analyzed may change frequently, the traditional solution requires manual data analysis for each change, and after obtaining the data format of the information to be analyzed, a matching formula is constructed for matching. The overall real-time difficulty and implementation cost are significantly higher.

[0031] This application automatically locates the location of the information to be crawled through injection for data collection, significantly improving the speed and accuracy of field acquisition and verification, is not affected by the uncertainty of backend data returned, reduces dependence on field naming rules, and improves the accuracy and reliability of data crawling.

[0032] In a preferred embodiment, the method of filling in simulation data with the data structure as a template to construct a simulation response includes: The data structure includes a plurality of data entries, each of which corresponds to a respective data value; The simulation data includes a simulation value of data including the unique identifier corresponding to each data category in the data structure; The simulated values ​​of the data are substituted for the original data values ​​to construct a simulated response.

[0033] In the embodiment of the present application, by analyzing the structure template of the response data and constructing simulated response data on this basis, the simulated data corresponds to the original data structure one by one, but replaces the actual value in the original data with a simulated value containing a unique identifier. In order to simulate the data response, it is ensured that a specific identifier can be obtained during the crawling process, thereby achieving more accurate data crawling and simulation. For example, Figure 3 As shown in the figure, it is a schematic diagram of the original response data without modification. The response data in JSON format is converted into a schema template. The data is reconstructed according to this schema template so that each field is different and has unique identification. The data of the replaced unique identifier can be as follows Figure 4 shown.

[0034] In a preferred embodiment, the method for obtaining the data interface of the web page backend corresponding to the marked position is: Obtaining the position coordinates of the marked position in the page to be analyzed; Use the positioning tool in the developer mode to locate the position coordinates and obtain the data interface at the coordinates; Get the URL information and request parameter information of the data interface.

[0035] In the embodiments of the present application, Figure 5 As shown in the figure, a schematic diagram of locating the location of a data interface at a specific location based on location coordinates is given. The network parameter data corresponding to the target location can be read through the built-in positioning tool in the browser, and then the URL information and request parameter information corresponding to the data interface can be obtained. Based on this solution, the acquisition is direct and convenient, which changes the traditional complex analysis method based only on data matching.

[0036] In a preferred embodiment, the method for obtaining the injected data string is: Configure a packet capture tool and set the packet capture tool as a system proxy; Sending a data request from the page end to the original target address, and intercepting the original request data based on the packet capture tool; Based on the data structure, the original request data is replaced by the simulated data in the simulated response to obtain the injected data string.

[0037] In the embodiment of the present application, the system proxy can be set up using a packet capture tool to intercept the target interface, and then breakpoints can be set in the packet capture tool to control and modify the data flow at a specific moment in the request process. When the user refreshes the page, the request to the original address API is intercepted, and the returned data is modified to the pre-constructed response data. The page will be rendered using the pre-constructed response data, and the real loading process will be run.

[0038] In a preferred embodiment, the method further comprises: Constructing a crawling template library, wherein the crawling template library is provided with a plurality of crawling templates, each of the crawling templates corresponds to a page to be analyzed, and each of the page to be analyzed is provided with a plurality of annotation positions; Obtain a preset crawling plan, which includes several crawling requirements and crawling strategies; Based on the crawling scheme, different crawling templates are called to crawl data.

[0039] In the embodiment of the present application, different crawling templates can be preset to deal with different web pages that need to collect data. In each template, different marked positions in different web pages are corresponding to different positions where data needs to be crawled. The crawling requirements in the crawling scheme can refer to crawling corresponding pages according to different user needs. The crawling strategy can specify strategies such as crawling at a certain time and crawling by number of times.

[0040] In a preferred embodiment, after obtaining the data interface of the webpage backend corresponding to the marked position, the method further includes: The format of the data interface is compared with the format of the standard data interface, and when the format comparison is wrong, a page layout change reminder is generated.

[0041] In the embodiment of the present application, since the change of the page text and image layout may cause the failure of reading the data at a specific location coordinate, when the format comparison is wrong, a page layout change reminder is generated to prompt the administrator to update the marked location data.

[0042] like Figure 6 As shown, in one embodiment, a data crawling device is provided, which can be integrated into the above-mentioned computer device 120, and specifically can include a marking location acquisition module 510, a data interface acquisition module 520, a response data acquisition module 530, a simulation response acquisition module 540, an injection page acquisition module 550 and a data crawling module 560.

[0043] The marked position acquisition module 510 is used to acquire a webpage screenshot of the page to be analyzed, mark the information to be crawled on the webpage screenshot, and obtain a number of marked positions; The data interface acquisition module 520 is used to acquire the data interface of the web page backend corresponding to the marked position; A response data acquisition module 530 is used to call the data interface and obtain response data from the data stream read from the data interface; A simulation response acquisition module 540 is used to analyze the response data, obtain a data structure of the response data, and use the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers; The injection page acquisition module 550 is used to send an initial data request from the page end to the original target address, intercept the response data string, inject the simulated response into the corresponding position in the response data string, and obtain the injection data string; the page end loads based on the injection data string to obtain the injection page; The data crawling module 560 is used to retrieve the unique identifier in the injected page, and then obtain the location of the data field corresponding to the information to be crawled in the back end of the web page, and crawl the information to be crawled based on the location.

[0044] In the embodiment of the present application, the explanation and description of the above-mentioned data crawling device can refer to the explanation and description of the above-mentioned corresponding method. For the description of the above-mentioned data crawling method, please refer to the above, which will not be repeated here.

[0045] In the embodiment of the present application, based on the present device, it is only necessary to mark the position of the information to be crawled on the page in the page, so that the position of the data at the marked position in the back-end data of the page can be quickly obtained, and the target data can be automatically crawled based on the position. The present application replaces the traditional scheme of first performing back-end data analysis and then constructing a matching formula for data acquisition with position annotation, and there is no need to perform a large amount of manual analysis and construct matching functions for each change of the back-end data of the page, which greatly saves the cost and difficulty of manual matching. In addition, considering that the layout of the page display does not change frequently, the data format, page structure or variable name of the back-end information to be analyzed may change frequently. In the traditional scheme, manual data analysis is required for each change, and the data format of the information to be analyzed is obtained and then the matching formula is constructed for matching, and the overall real-time difficulty and implementation cost are obviously higher. The present application automatically locates the position of the information to be crawled by injection for data collection, significantly improves the speed and accuracy of field acquisition and verification, is not affected by the uncertainty of the back-end data return, reduces the dependence on the field naming rules, and improves the accuracy and reliability of data crawling.

[0046] In one embodiment, the apparatus further comprises: The page layout change reminder module is used to compare the format of the data interface with the format of the standard data interface, and generate a page layout change reminder when the format comparison is wrong.

[0047] In an embodiment of the present application, the page layout change reminder module can promptly remind the administrator or user to update the coordinates of the marked location.

[0048] Figure 7 The internal structure diagram of a computer device in one embodiment is shown. The computer device may specifically be Figure 1 The terminal 110 in FIG. Figure 7As shown, the computer device includes a processor, a memory, a network interface, an input device and a display screen connected through a system bus. Among them, the memory includes a non-volatile storage medium and an internal memory. The non-volatile storage medium of the computer device stores an operating system and may also store a computer program. When the computer program is executed by the processor, the processor can implement a data crawling method. The internal memory may also store a computer program. When the computer program is executed by the processor, the processor can execute a data crawling method. The display screen of the computer device may be a liquid crystal display or an electronic ink display screen, and the input device of the computer device may be a touch layer covering the display screen, or a button, trackball or touchpad provided on the housing of the computer device, or an external keyboard, touchpad or mouse, etc.

[0049] Those skilled in the art will understand that Figure 6 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0050] In one embodiment, the data crawling device provided by the present application can be implemented in the form of a computer program. The computer program can be Figure 7 The computer device shown in the figure is run. The memory of the computer device can store various program modules constituting the data crawling device, for example, Figure 6 The illustrated annotation location acquisition module 510, data interface acquisition module 520, response data acquisition module 530, simulation response acquisition module 540, injection page acquisition module 550 and data crawling module 560. The computer program composed of each program module enables the processor to execute the steps of the data crawling method of each embodiment of the present application described in this specification.

[0051] For example, Figure 7 The computer device shown can be Figure 6 The marked position acquisition module 510 in the data crawling device shown executes step S10. The computer device can execute step S20 through the data interface acquisition module 520. And so on.

[0052] In one embodiment, a data crawling system is proposed, including a memory and a processor, wherein a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the data crawling method as described above.

[0053] In the embodiments of the present application, please refer to the above for the description of the above-mentioned data crawling method, which will not be repeated here.

[0054] In the embodiment of the present application, it is only necessary to mark the position of the information to be crawled on the page, so that the position of the data at the marked position in the back-end data of the page can be quickly obtained, and the target data can be automatically crawled based on the position. The present application replaces the traditional scheme of first performing back-end data analysis and then constructing a matching formula for data acquisition with position annotation, and there is no need to perform a large amount of manual analysis and construct matching functions for each change of the back-end data of the page, which greatly saves the cost and difficulty of manual matching. In addition, considering that the layout of the page display does not change frequently, the data format, page architecture or variable name of the back-end information to be analyzed may change frequently. In the traditional scheme, manual data analysis is required for each change, and the data format of the information to be analyzed is obtained and then the matching formula is constructed for matching. The overall real-time difficulty and implementation cost are obviously higher. The present application automatically locates the position of the information to be crawled by injection for data collection, significantly improves the speed and accuracy of field acquisition and verification, is not affected by the uncertainty of the back-end data return, reduces the dependence on the field naming rules, and improves the accuracy and reliability of data crawling.

[0055] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the processor executes the steps of the data crawling method as described above.

[0056] In the embodiments of the present application, please refer to the above for the description of the above-mentioned data crawling method, which will not be repeated here.

[0057] In the embodiment of the present application, the program run by the method stored in the storage medium of the embodiment of the present application only needs to mark the position of the information to be crawled on the page, so that the position of the data at the marked position in the back-end data of the page can be quickly obtained, and the target data can be automatically crawled based on the position. The present application replaces the traditional scheme of first performing back-end data analysis and then constructing a matching formula for data acquisition with position annotation, and there is no need to perform a large amount of manual analysis and construct matching functions for each change of the back-end data of the page, which greatly saves the cost and difficulty of manual matching. In addition, considering that the layout of the page display does not change frequently, the data format, page architecture or variable name of the back-end information to be analyzed may change frequently. In the traditional scheme, manual data analysis is required for each change, and the data format of the information to be analyzed is obtained and then the matching formula is constructed for matching, and the overall real-time difficulty and implementation cost are obviously higher. The present application automatically locates the position of the information to be crawled by injection for data collection, significantly improves the speed and accuracy of field acquisition and verification, is not affected by the uncertainty of the back-end data return, reduces the dependence on the field naming rules, and improves the accuracy and reliability of data crawling.

[0058] It should be understood that, although each step in the flow chart of each embodiment of the present application is shown in sequence according to the indication of the arrow, these steps are not necessarily performed in sequence according to the order indicated by the arrow. Unless there is clear explanation in this article, the execution of these steps does not have strict order restriction, and these steps can be performed in other orders. Moreover, at least a portion of the steps in each embodiment may include a plurality of sub-steps or a plurality of stages, and these sub-steps or stages are not necessarily performed at the same time, but can be performed at different times, and the execution order of these sub-steps or stages is not necessarily performed in sequence, but can be performed in turn or alternately with at least a portion of other steps or sub-steps or stages of other steps.

[0059] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the program can be stored in a non-volatile computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0060] The technical features of the above-described embodiments may be arbitrarily combined. To make the description concise, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0061] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A data crawling method, characterized in that: The method comprises: Obtain a webpage screenshot of the page to be analyzed, and mark the information to be crawled on the webpage screenshot to obtain a number of marked positions; Obtaining the data interface of the webpage backend corresponding to the marked position; Calling the data interface to obtain response data from the data stream read from the data interface; Analyze the response data to obtain a data structure of the response data, and use the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers; Sending a data request from the page end to the original target address, intercepting a response data string, injecting the simulated response into a corresponding position in the response data string, and obtaining an injected data string; the page end performs loading based on the injected data string to obtain an injected page; The unique identifier in the injected page is retrieved, and then the location of the data field corresponding to the information to be crawled in the back end of the web page is obtained, and data crawling is performed on the information to be crawled based on the location.

2. A data crawling method according to claim 1, characterized in that: The method of filling in simulation data with the data structure as a template to construct a simulation response includes: The data structure includes a plurality of data entries, each of which corresponds to a respective data value; The simulation data includes a simulation value of data including the unique identifier corresponding to each data category in the data structure; The simulated values ​​of the data are substituted for the original data values ​​to construct a simulated response.

3. A data crawling method according to claim 1, characterized in that: The method for obtaining the data interface of the web page backend corresponding to the marked position is: Obtaining the position coordinates of the marked position in the page to be analyzed; Use the positioning tool in the developer mode to locate the location coordinates and obtain the data interface at the coordinates; Get the URL information and request parameter information of the data interface.

4. A data crawling method according to claim 1, characterized in that: The method for obtaining the injected data string is: Configure a packet capture tool and set the packet capture tool as a system proxy; Sending a data request from the page end to the original target address, and intercepting the original request data based on the packet capture tool; Based on the data structure, the original request data is replaced by the simulated data in the simulated response to obtain the injected data string.

5. A data crawling method according to claim 1, characterized in that: The method further comprises: Constructing a crawling template library, wherein the crawling template library is provided with a plurality of crawling templates, each of the crawling templates corresponds to a page to be analyzed, and each of the page to be analyzed is provided with a plurality of annotation positions; Obtain a preset crawling plan, which includes several crawling requirements and crawling strategies; Based on the crawling scheme, different crawling templates are called to crawl data.

6. A data crawling method according to claim 1, characterized in that: After obtaining the data interface of the webpage backend corresponding to the marked position, the method further includes: The format of the data interface is compared with the format of the standard data interface, and when the format comparison is wrong, a page layout change reminder is generated.

7. A data crawling device, characterized in that: The data crawling device comprises: The marked position acquisition module is used to obtain a webpage screenshot of the page to be analyzed, mark the information to be crawled on the webpage screenshot, and obtain a number of marked positions; A data interface acquisition module, used to acquire the data interface of the web page backend corresponding to the marked position; A response data acquisition module, used for calling the data interface and obtaining response data from the data stream read from the data interface; A simulation response acquisition module, used for analyzing the response data, obtaining a data structure of the response data, and using the data structure as a template to fill in simulation data to construct a simulation response, wherein the simulation data includes a number of unique identifiers; The injection page acquisition module is used to send an initial data request from the page end to the original target address, intercept the response data string, inject the simulated response into the corresponding position in the response data string, and obtain the injection data string; the page end is loaded based on the injection data string to obtain the injection page; The data crawling module is used to retrieve the unique identifier in the injected page, and then obtain the location of the data field corresponding to the information to be crawled in the back end of the web page, and crawl the information to be crawled based on the location.

8. A data crawling device according to claim 7, characterized in that: The device also includes: The page layout change reminder module is used to compare the format of the data interface with the format of the standard data interface, and generate a page layout change reminder when the format comparison is wrong.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the steps of the data crawling method described in any one of claims 1 to 6.

10. A data crawling system, characterized in that: It includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the data crawling method as described in any one of claims 1 to 6.

Images