A data grabbing method and device, computer equipment and storage medium

Abstract

The application discloses a data crawling method and device, computer equipment and a storage medium. The method receives crawling configuration information, the crawling configuration information including a URL of a target crawling page and target crawling content; the target crawling content is parsed to generate a plurality of crawling tasks corresponding to independent data fields; based on the URL of the target crawling page loading a page, data crawling is performed on each crawling task to obtain DOM data and response data of a network request; the DOM data and the response data of the network request are parsed to obtain structured crawling data of the crawling task. The efficiency of data crawling can be improved.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, and in particular to a data capture method, apparatus, computer equipment, and storage medium. Background Technology

[0002] In the information age, data acquisition and processing play a crucial role for various enterprises and organizations. Current data scraping technologies primarily rely on manually written scripts. However, manually writing scripts requires highly skilled programmers, necessitating significant time and effort, resulting in low development efficiency and limiting the timeliness of data acquisition.

[0003] When the target website changes, existing scripts need to be manually debugged and modified, making it difficult to guarantee the continuous and stable operation of data scraping. Currently, some simple data scraping tools exist that can perform simple, standardized scraping tasks, but they struggle to handle complex data scraping tasks. Summary of the Invention

[0004] Therefore, embodiments of this application provide a data crawling method, apparatus, computer device, and storage medium, which realizes automatic data identification and crawling of target web pages, thereby improving the efficiency of data crawling.

[0005] Firstly, this application provides a data scraping method.

[0006] This application is achieved through the following technical solution: A data scraping method, comprising: S1: Receive crawling configuration information, which includes the URL of the target crawling page and the target crawling content; S2: Parse the target content to generate several crawling tasks with corresponding independent data fields; S3: Load the page based on the URL of the target crawling page, perform data crawling for each crawling task, and obtain DOM data and network request response data; S4: Parse the DOM data and the response data of the network request to obtain structured crawling data for the crawling task.

[0007] In a preferred example of this application, it may further be configured to include: S5: When there are incomplete crawling tasks, identify the deep detail links in the target crawling page, jump to the deep detail links to execute steps S3 to S4, and obtain deep crawling data; S6: Merge deep crawled data with structured crawled data to obtain the final crawled data.

[0008] In a preferred example of this application, it can be further configured to perform data scraping for each scraping task, obtaining DOM data and response data from network requests, specifically including: For each crawling task, a dual-path data crawling is performed. The first path retrieves the DOM data of the target crawling page, and the second path retrieves the response data of the network requests intercepted and cached by the local proxy server.

[0009] In a preferred example of this application, it can be further configured to receive capture configuration information, including: Receive user-inputted capture configuration information in a preset format; The parameter module parses the capture configuration information; The parameter module is converted into the URL of the executable target crawl page and the target crawl content corresponding to each target crawl page.

[0010] In a preferred example of this application, the target crawling content can be further configured to be parsed to generate a crawling task with several corresponding independent data fields, including: Based on preset prompts, the target content described in natural language is split into several data fields using a pre-trained large language model, with each crawling task corresponding to an independent data field. Assign a unique task identifier to each crawling task and establish the association between the crawling task and the data field.

[0011] In a preferred example of this application, the DOM data and the response data can be further configured to parse the data to obtain structured crawling data for the crawling task, including: Use CSS selectors to extract the text content from the DOM data to obtain the first parsing result; Use JSONPath to query the nested data in the response data to obtain the second parsing result; The first and second parsing results are combined to form structured crawling data for the crawling task; Bind structured crawled data to task identifiers.

[0012] In a preferred embodiment of this application, it can be further configured to include: monitoring the task status code of the crawling task; if the task status code of the crawling task indicates that the crawling task is in an abnormal state, executing a retry strategy, wherein, When the abnormal state is due to an abnormal network environment, the system will retry the preset number of times according to the exponential backoff strategy. Alternatively, when the abnormal state is a page construction problem, update the parsing rules and perform data crawling based on the new parsing rules; Alternatively, when the abnormal status is a data parsing problem, perform a preset number of retries for parsing; Alternatively, when the abnormal status is a storage problem, perform a preset number of storage retries.

[0013] Secondly, this application provides a data capture device.

[0014] This application is achieved through the following technical solution: A data scraping apparatus for performing the data scraping method described in the first aspect above, the apparatus comprising: The configuration information receiving module is used to receive crawling configuration information, which includes the URL of the target crawling page and the target crawling content; The task decomposition module is used to parse the target crawling content and generate several crawling tasks with corresponding independent data fields. The data crawling module is used to load pages based on the URL of the target crawling page, perform data crawling for each crawling task, and obtain DOM data and response data of network requests; The data parsing module is used to parse the DOM data and the response data of the network request to obtain structured crawling data for the crawling task.

[0015] Thirdly, this application is achieved through the following technical solution: A computer device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the above-described data capture methods.

[0016] Fourthly, this application provides a computer-readable storage medium.

[0017] This application is achieved through the following technical solution: A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of any of the above-described data capture methods.

[0018] In summary, compared with the prior art, the beneficial effects of the technical solution provided in this application include at least the following: This application receives crawling configuration information, including the URL of the target crawling page and the target crawling content; parses the target crawling content to generate several crawling tasks with corresponding independent data fields; loads the page based on the URL of the target crawling page; performs data crawling on each crawling task to obtain DOM data and network request response data; and parses the DOM data and the network request response data to obtain structured crawling data for the crawling tasks. By receiving crawling requests through a visual configuration interface, automatically parsing the crawling content into multiple independent crawling tasks, and automatically crawling both types of data for each task, the application significantly improves data crawling efficiency and ensures timely data acquisition. Attached Figure Description

[0019] Figure 1 A schematic flowchart of a data capture method provided in an embodiment of this application; Figure 2 A flowchart illustrating a data capture method provided in yet another embodiment of this application; Figure 3 A flowchart illustrating a data capture method provided in another embodiment of this application; Figure 4 A schematic flowchart of a data capture method provided in an embodiment of this application; Figure 5 This is a schematic diagram of the structure of a data capture device provided in an embodiment of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application.

[0020] Explanation of reference numerals in the attached figures: Configuration information receiving module 10, task decomposition module 20, data capture module 30, and data parsing module 40. Detailed Implementation

[0021] This specific embodiment is merely an explanation of this application and is not intended to limit it. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of this application.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0023] Furthermore, the term "and / or" in this application is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this application, unless otherwise specified, generally indicates that the preceding and following related objects have an "or" relationship.

[0024] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0025] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0026] In the information age, data acquisition and processing play a crucial role for various enterprises and organizations. Current data scraping technologies primarily rely on manually written scripts, which require a high level of programming skill from technical personnel. Script writing demands significant time and effort, resulting in low development efficiency and limiting the timeliness of data acquisition. Furthermore, when the target website changes—for example, when the target address's data structure or webpage structure changes or is updated—existing scripts need to be manually debugged and modified, making it difficult to guarantee the continuous and stable operation of data scraping.

[0027] The embodiments of this application will now be described in further detail with reference to the accompanying drawings.

[0028] The first exemplary embodiment of this application provides a data scraping method applicable to scenarios such as data collection and public opinion monitoring. The executing entity of this data scraping method can be the data scraping device provided in the embodiments of this application, or an electronic device integrating the aforementioned data scraping device. The electronic device can be a smartphone, tablet computer, laptop computer, or desktop computer, etc.

[0029] See Figure 1 As shown, the specific steps of the data scraping method include: S1: Receive crawl configuration information, which includes the URL of the target crawl page and the target crawl content.

[0030] The system accepts user input via a terminal device to configure the crawling settings. This settings specify the content the user wants to crawl, including at least the URL of the target page and fields containing the target content. Specifically, users can input the crawling settings through the interactive interface of their terminal device. The URL is the address used to uniquely identify and locate a webpage, including information such as the protocol, hostname, and path. The target content field is a natural language description of the content to be crawled, such as the title, body, comment data, author information, and contact information of a post.

[0031] For example, if a user needs to scrape information from the Top 50 movies on Douban, the system receives input from the user via their terminal device: Target scraping page: https: / / movie.douban.com / top50, Target scraping content: "I need to scrape the ranking, Chinese name, English name, director, lead actors (top two), release year, rating, and number of ratings for each movie."

[0032] S2: Parse the target content fields to generate several crawling tasks with corresponding independent data fields.

[0033] In practical implementation, NLP (Natural Language Processing) technology can be used to identify the natural language description requirements in the target content fields. NLP libraries can be used to analyze sentence structure for basic word segmentation, identify key entities, and the system can generate a list of fields that conforms to a preset structure. A crawling task can then be generated for each independent field in the field list.

[0034] S3: Load the page based on the URL of the target crawl page, perform data crawling for each crawling task, and obtain DOM data and response data of network requests.

[0035] It should be noted that the DOM is a structured, tree-like in-memory object model that the browser parses from downloaded HTML. DOM data represents the state and content of all nodes in this tree structure at the current moment, such as elements, attributes, and text. Network request response data is the raw response content returned by the server after the browser sends all HTTP / HTTPS requests to the server while loading a webpage.

[0036] S4: Parse the DOM data and network request response data to obtain structured crawling data for the crawling task.

[0037] In this process, the DOM data obtained in step S3 is converted into Markdown format using a conversion tool (such as Pandoc), and the Markdown format is parsed to determine the core data that the crawling task currently being executed needs to collect.

[0038] For processing the response data, the data format is first determined, and data is extracted using different query languages ​​according to the crawling task configuration. The extracted raw data is then converted into the target data type, and the processed data is assembled into structured crawled data.

[0039] In one embodiment, see Figure 2 As shown, this data scraping method also includes: S5: When there are incomplete crawling tasks, identify the deep detail links in the target crawling page, jump to the deep detail links to execute steps S3 to S4, and obtain the deep crawling data; S6: Merge deep crawled data with structured crawled data to obtain the final crawled data.

[0040] In practice, during data scraping, it's necessary to monitor the completion status of the scraping task. This can be achieved by checking the task status code to determine success, failure, or any abnormalities. If the task is incomplete, further identification of deeper detail links on the target page is required. Deeper detail links are URLs on the target page that point to more detailed content pages. Redirecting to these links allows for data scraping and parsing, yielding the deeper scraped data. The task status code is an identifier representing the execution status of the scraping task. Task status codes can be numeric, string, or a combination of both.

[0041] The deep crawled data is then filled into the structured crawled data to obtain the final crawled data.

[0042] In practice, the final crawled data also needs to be deduplicated and optimized.

[0043] This application can receive data scraping requests through a visual configuration interface, automatically parse the scraped content into multiple independent scraping tasks, and automatically perform dual-path scraping for each scraping task, thereby greatly improving data scraping efficiency and ensuring the timeliness of data acquisition.

[0044] In one exemplary embodiment, see Figure 3 As shown, each crawling task performs data crawling to obtain DOM data and network request response data, specifically including: For each crawling task, a dual-path data crawling is performed. The first path retrieves the DOM data of the target crawling page, and the second path retrieves the response data of the network requests intercepted and cached by the local proxy server.

[0045] The first path is suitable for crawling visible data directly embedded in HTML. Specifically, a headless browser is used to load the page according to the URL of the target crawl page. After the basic content has been loaded, for each crawling task, CSS selectors are used to identify the core data of the currently executing crawling task contained in the target crawl page. The core data is the DOM data of the currently executing crawling task.

[0046] The second approach is suitable for fetching dynamically loaded data. Specifically, it starts a local proxy server to listen, registers response callback functions to capture network requests, intercepts the response data returned by the network requests, and caches the response data.

[0047] It should be noted that, firstly, a local proxy listener is created in the crawler program to act as a communication intermediary between the browser and the target website, responsible for forwarding network traffic. The response callback function refers to the event handling function registered on the local proxy server, which is custom logic automatically triggered when the proxy receives a network response. A network request is a data retrieval instruction sent by the client to the server, including the request method, URL, headers, and body. Response data refers to the content returned by the local proxy server in response to the network request, including the status code, headers, and body data.

[0048] In one exemplary embodiment, receiving capture configuration information includes: Receive user-inputted capture configuration information in a preset format; The module for parsing and capturing parameters from configuration information; The parameter module is converted into executable target crawl page URLs and corresponding target crawl content for each target crawl page.

[0049] Specifically, it can receive capture configuration information filled in by users through web forms or command-line tools. The capture configuration information can be predefined input with specific data structures and fields, and the preset formats include JSON, YAML, XML and other formats. The corresponding parser reads the configuration information and parses out the parameter modules that define the crawling behavior, i.e., the logical components, including but not limited to the base URL, pagination rules, link extraction rules, target data definition, request parameters, and processing logic. When the crawling configuration information is in JSON format, a standard JSON parsing library is used, such as Python's `json` module; when the crawling configuration information is in YAML format, a YAML parsing library is used, such as Python's `PyYAML` module; and when the crawling configuration information is in XML format, an XML parsing library is used, such as Python's `xml.etree.ElementTree` module. Then, based on the parameter modules, the specific target crawling pages for the crawling task and the target crawling content corresponding to each target crawling page are generated, clearly indicating the range of data to be crawled for each target crawling page.

[0050] In one exemplary embodiment, the target crawling content is parsed to generate a crawling task with several corresponding independent data fields, including: Based on preset prompts, a pre-trained large language model is used to split the target content described in natural language into several data fields, with each crawling task corresponding to an independent data field; a unique task identifier is assigned to each crawling task, and the association between the crawling task and the data field is established.

[0051] The pre-training process for the large language model involves: constructing a large number of natural language crawling requirements, labeling the corresponding structured field lists for these natural language crawling requirements, training the open-source large language model using the natural language crawling requirements and labeled data, verifying whether the structure generated by the large language model meets the requirements, and continuing until the performance of the large language model meets the requirements.

[0052] Specifically, the user enters their target content for scraping, "I need the names, prices, and image links of all products on this page," through a text box. This input is then concatenated with a preset prompt to form the complete input. The preset prompt could be something like, "Please break down the following user scraping requirements into independent data fields and output a list of field names in JSON array format." The complete input is sent to a pre-trained LLM server, which outputs a structured list of fields ["product_name", "price", "image_url"]. An independent scraping task is created for each data field, and a task identifier (e.g., UUID v4) is assigned to each task. The mapping relationship between scraping tasks, data fields, and task identifiers is stored.

[0053] In one exemplary embodiment, DOM data and response data are parsed to obtain structured crawling data for the crawling task, including: Use CSS selectors to extract text content from DOM data to obtain the first parsing result; Use JSONPath to query nested data in the response data to obtain a second parsed result; The first and second parsing results are combined to form structured crawling data for the crawling task; Bind structured crawled data to task identifiers.

[0054] Specifically, basic information is extracted from the HTML DOM data of the target page using CSS selectors, serving as the first parsing result; dynamic information is extracted from the AJAX response using JSONPath, serving as the second parsing result; these two parts of data are combined to form structured data; and the structured data is bound to a pre-assigned task identifier. This dual-path parsing architecture can capture more complete data.

[0055] In one exemplary embodiment, identifying deep detail links in the target crawled page includes: Identify and crawl all instances of the same domain with href redirects on the target page. The `data-detail-url` attribute in the tag matches URL paths that conform to preset regular expression rules.

[0056] Furthermore, the threshold for deep detail links is set to three levels.

[0057] Specifically, there are a maximum of three levels of deep detail links. First, if the required data is obtained from the first-level deep detail links on the target page, the process ends. If not, the process continues by identifying deeper detail links within the first level and marking them as second-level deep detail links. If the required data is obtained from these second-level links, the process ends. If not, the process continues by identifying deeper detail links within the second level and marking them as third-level deep detail links. If the required data is obtained from these third-level links, the process ends, and this data is considered deep detail data. If the required data is still not obtained from these third-level deep detail links, the crawling task is marked as not containing the required data.

[0058] In some exemplary embodiments, the data capture method provided in this application further includes: monitoring the task status code of the capture task, and executing a retry strategy if the task status code of the capture task indicates that the capture task is in an abnormal state.

[0059] Specifically, if the task status code of a crawling task indicates that the crawling task is in an abnormal state, a retry strategy is executed, including: When the abnormal state is due to a network environment anomaly, the system will retry a preset number of times according to the exponential backoff strategy. Specifically, the preset number of retries is set to three: the first retrieval waits for 1 second, the second for 2 seconds, and the third for 3 seconds. In actual implementation, the current network connection status is checked before each retrieval to ensure that the network has returned to normal before proceeding with the retry.

[0060] Alternatively, when the abnormal state is a page construction problem, update the parsing rules and perform data crawling based on the new parsing rules.

[0061] Specifically, when a structural change to the target webpage causes the data crawling process to malfunction, the page needs to be re-parsed to determine the cause of the original locator's failure. After correction, a new locator is used to crawl the data. If the crawling still fails after a preset number of retries, an alarm mechanism is used to send the cause of the failure to the corresponding client of the developers. This allows the developers to analyze and correct the cause of the anomaly in a timely manner, ensuring the smooth progress of data crawling.

[0062] Alternatively, when the abnormal status is a data parsing problem, perform a preset number of retries for parsing.

[0063] Specifically, when the required data is found to be unavailable or not loading on the target page, an automatic retry operation will be performed a preset number of times (e.g., 3 times) within a preset time. If the required data still does not exist after the retry, the field will be marked as missing, and the crawling process will continue; for missing data, intelligent completion can be performed based on historical version data or related field logic.

[0064] Alternatively, when the abnormal status is a storage problem, perform a preset number of storage retries.

[0065] Specifically, if a stored procedure encounters a problem, such as a database connection failure, it attempts to reconnect and re-execute the storage operation. After a storage failure, the data is temporarily cached in a local file and re-uploaded to the storage system once it recovers. A backup storage system is also configured; if the storage problem persists, the backup storage system is invoked, and the fetched data is stored there. In actual implementation, detailed logs must be recorded when storage problems occur for developers to analyze the issue.

[0066] In one feasible implementation, see Figure 4 As shown, a data scraping method according to this application includes the following steps: First, the system receives basic information about the crawling task from the user. It then uses AI (NLP) to parse the natural language task description, breaking it down into multiple tasks. For each task, it executes two data crawling paths: the first path identifies DOM content potentially containing core data, converts the target HTML DOM content into Markdown syntax, identifies the Markdown syntax, parses and collects potentially core data; the second path intercepts request and response data through a local proxy server, integrates and stores it in a database, parses the core data, and combines the core data from both paths. Data supplementation is then performed. If the current page cannot fully identify core data, deep links are identified and their redirection paths are recorded, and the link content is categorized. Based on the link categorization and parameters, deep links are constructed to avoid repeatedly crawling shallow pages and redirecting. The crawled data is further deduplicated and summarized. The backend establishes a unified data reporting interface to receive, store, and analyze data, establishes a unified alarm mechanism and heartbeat detection, sets execution frequency to monitor execution status, and addresses data crawling failures. When an error occurs, the system identifies error message keywords to determine the error type. If the error type is a network problem, it executes network-related retry strategies, such as retries according to preset numbers, intervals, and methods. If the error type is a page structure problem, it executes page structure-related retry strategies. If the error type is a data parsing problem, it executes data parsing-related retry strategies. If the error type is a data storage problem, it executes data storage-related retry strategies. When the maximum number of retries is reached, the system automatically identifies and re-fetches the page structure, and the AI-generated task generates a complete workflow description. It automatically identifies and controls the fetch task version, records execution status and results, and calculates execution time and accuracy. Users can compare different versions of the fetch task through version control to optimize execution efficiency and accuracy.

[0067] Another embodiment of this application also provides a data scraping device for performing the above-described data scraping method, such as... Figure 5 As shown, the device includes: The configuration information receiving module 10 is used to receive crawling configuration information, which includes the URL of the target crawling page and the target crawling content; The task decomposition module 20 is used to parse the target crawling content and generate several crawling tasks with corresponding independent data fields. The data crawling module 30 is used to load the page based on the URL of the target crawling page, perform data crawling for each crawling task, and obtain DOM data and response data of network requests; The data parsing module 40 is used to parse DOM data and network request response data to obtain structured crawling data for the crawling task.

[0068] The specific limitations of the data capture device provided in this embodiment can be found in the embodiments of the data capture method described above, and will not be repeated here. Each module in the above-described data capture device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0069] This specification also provides a computer-readable storage medium storing a computer program that, when executed by a processor, can be used to perform the data capture method described above.

[0070] The embodiments in this specification also provide Figure 6 The diagram shows the structure of the electronic device. Figure 6 At the hardware level, the electronic device includes a processor, an internal bus, a network interface, memory, and non-volatile storage, and may also include other hardware required for the business. The processor reads the corresponding computer program from the non-volatile storage into memory and then runs it to implement the data capture method described in the first embodiment above.

[0071] Of course, in addition to software implementation, this specification does not exclude other implementation methods, such as logic devices or a combination of hardware and software. In other words, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0072] Improvements in a technology can be clearly distinguished as either hardware improvements (e.g., improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the methodology). However, with technological advancements, many improvements to the methodology can now be considered direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved methodology into the hardware circuit. Therefore, it cannot be said that an improvement in methodology cannot be implemented using hardware physical modules. For example, a Programmable Logic Device (PLD) (e.g., a Field Programmable Gate Array (FPGA)) is such an integrated circuit whose logic function is determined by the user programming the device. Designers can program and "integrate" a digital system onto a PLD themselves, without needing chip manufacturers to design and manufacture dedicated integrated circuit chips. Furthermore, nowadays, instead of manually manufacturing integrated circuit chips, this programming is mostly implemented using "logic compiler" software. Similar to the software compiler used in program development, the original code before compilation must be written in a specific programming language, called a Hardware Description Language (HDL). There are many HDLs, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, and RHDL (Ruby Hardware Description Language). Currently, VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog are the most commonly used. Those skilled in the art should understand that by simply performing some logic programming on the method flow using one of these hardware description languages ​​and programming it into an integrated circuit, the hardware circuit implementing the logical method flow can be easily obtained.

[0073] The controller can be implemented in any suitable manner. For example, it can take the form of a microprocessor or processor and a computer-readable medium storing computer-readable program code (e.g., software or firmware) executable by the (micro)processor, logic gates, switches, application-specific integrated circuits (ASICs), programmable logic controllers, and embedded microcontrollers. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicon Labs C8051F320. A memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art will also recognize that, in addition to implementing the controller in purely computer-readable program code form, the same functionality can be achieved by logically programming the method steps to make the controller take the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, such a controller can be considered a hardware component, and the means included therein for implementing various functions can also be considered as structures within the hardware component. Alternatively, the means for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0074] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, a computer can be, for example, a personal computer, laptop computer, cellular phone, camera phone, smartphone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or any combination of these devices.

[0075] For ease of description, the above devices are described in terms of function, divided into various units. Of course, in implementing this specification, the functions of each unit can be implemented in one or more software and / or hardware components.

[0076] Those skilled in the art will understand that embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0077] This specification is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this specification. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create a machine for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0078] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0079] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0080] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0081] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0082] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0083] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0084] Those skilled in the art will understand that the embodiments of this specification can be provided as methods, systems, or computer program products. Therefore, this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this specification may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0085] This specification can be described in the general context of computer-executable instructions that are executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This specification can also be practiced in distributed computing environments, where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0086] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to interchangeably. Each embodiment focuses on describing the differences from other embodiments. In particular, the system embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments.

[0087] The above description is merely an embodiment of this specification and is not intended to limit this specification. Various modifications and variations can be made to this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims of this specification.

Claims

1. A data scraping method, characterized in that, The method includes: S1: Receive crawling configuration information, which includes the URL of the target crawling page and the target crawling content; S2: Parse the target content to generate several crawling tasks with corresponding independent data fields; S3: Load the page based on the URL of the target crawling page, perform data crawling for each crawling task, and obtain DOM data and response data of network requests; S4: Parse the DOM data and the response data of the network request to obtain structured crawling data for the crawling task.

2. The data capture method according to claim 1, characterized in that, Also includes: S5: When there are incomplete crawling tasks, identify the deep detail links in the target crawling page, jump to the deep detail links to execute steps S3 to S4, and obtain deep crawling data; S6: Merge deep crawled data with structured crawled data to obtain the final crawled data.

3. The data capture method according to claim 1, characterized in that, For each crawling task, data is crawled to obtain DOM data and network request response data, specifically including: For each crawling task, a dual-path data crawling is performed. The first path retrieves the DOM data of the target crawling page, and the second path retrieves the response data of the network requests intercepted and cached by the local proxy server.

4. The data capture method according to claim 1, characterized in that, Receive capture configuration information, including: Receive user-inputted capture configuration information in a preset format; The parameter module in the capture configuration information is parsed; The parameter module is converted into the URL of the executable target crawl page and the target crawl content corresponding to each target crawl page.

5. The data capture method according to claim 1, characterized in that, The target content is parsed to generate several crawling tasks with corresponding independent data fields, including: Based on preset prompts, the target content described in natural language is split into several data fields using a pre-trained large language model, with each crawling task corresponding to an independent data field. Assign a unique task identifier to each crawling task and establish the association between the crawling task and the data field.

6. The data capture method according to claim 1, characterized in that, The DOM data and the response data are parsed to obtain structured crawling data for the crawling task, including: Use CSS selectors to extract the text content from the DOM data to obtain the first parsing result; Use JSONPath to query the nested data in the response data to obtain the second parsing result; The first and second parsing results are combined to form structured crawling data for the crawling task; Bind structured crawled data to task identifiers.

7. The data scraping method according to any one of claims 1 to 6, characterized in that, It also includes: monitoring the task status code of the crawling task; if the task status code indicates that the crawling task is in an abnormal state, executing a retry strategy, wherein... When the abnormal state is due to an abnormal network environment, the system will retry the preset number of times according to the exponential backoff strategy. Alternatively, when the abnormal state is a page construction problem, update the parsing rules and perform data crawling based on the new parsing rules; Alternatively, when the abnormal status is a data parsing problem, perform a preset number of retries for parsing; Alternatively, when the abnormal status is a storage problem, perform a preset number of storage retries.

8. A data acquisition device, characterized in that, The apparatus for performing the data capture method as described in claim 1 includes: The configuration information receiving module is used to receive crawling configuration information, which includes the URL of the target crawling page and the target crawling content; The task decomposition module is used to parse the target crawling content and generate several crawling tasks with corresponding independent data fields. The data crawling module is used to load pages based on the URL of the target crawling page, perform data crawling for each crawling task, and obtain DOM data and response data of network requests; The data parsing module is used to parse the DOM data and the response data of the network request to obtain structured crawling data for the crawling task.

9. A computer device, characterized in that, The method includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method according to any one of claims 1 to 7.

Images