Crawler configuration method and related device
The web crawler configuration method, which features a visual interface and real-time debugging capabilities, solves the problem of non-professional users struggling to configure web crawlers, enabling fast and accurate web crawler source configuration and improving the user experience.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING HONGTENG INTELLIGENT TECH CO LTD
- Filing Date
- 2024-12-02
- Publication Date
- 2026-06-02
Smart Images
Figure CN122132613A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the fields of computer and communication technology, and more specifically, to a web crawler configuration method and related equipment. Background Technology
[0002] Targeted web crawling platforms serve analysts by collecting security-related information, virus samples, and content posted by specific users (blogs, tweets, etc.) from the internet, followed by secondary processing and distribution. Existing crawling systems require technical personnel to write their own programs; some employ visual methods, but configuration still requires expertise in the relevant field. Summary of the Invention
[0003] The embodiments of this application provide a web crawler configuration method and related equipment, which enables users who have a basic understanding or even no understanding of web crawler principles to quickly configure web crawler sources.
[0004] Other features and advantages of this application will become apparent from the following detailed description, or may be learned in part from practice of this application.
[0005] According to one aspect of the embodiments of this application, a web crawler configuration method is provided, including:
[0006] In response to the selection of the configuration mode, the corresponding crawler configuration interface is displayed, which contains multiple crawler configuration items.
[0007] In response to the configuration operations on each of the crawler configuration items, a preliminary configuration scheme is obtained;
[0008] In response to the triggering of the debugging function of the configuration interface, the corresponding debugging steps are executed according to the preliminary configuration scheme;
[0009] Upon completion of debugging, the final configuration scheme is formed.
[0010] According to one aspect of the embodiments of this application, a web crawler configuration device is provided, applied to a web crawler configuration system including a visual web crawler configuration interface, the web crawler configuration device comprising:
[0011] The interface display module is used to display the corresponding crawler configuration interface in response to the selection of the configuration mode. The crawler configuration interface contains multiple crawler configuration items.
[0012] The preliminary configuration module is used to obtain a preliminary configuration scheme in response to the configuration operations of each of the crawler configuration items;
[0013] The step debugging module is used to respond to the triggering of the debugging function of the configuration interface and execute the corresponding debugging steps according to the preliminary configuration scheme.
[0014] The scheme formation module is used to generate the final configuration scheme in response to the completion of debugging.
[0015] According to one aspect of the embodiments of this application, a computer-readable medium is provided having a computer program stored thereon, which, when executed by a processor, implements the crawler configuration method as described in the above embodiments.
[0016] According to one aspect of the embodiments of this application, an electronic device is provided, including: one or more processors; and a storage device for storing one or more programs, which, when executed by the one or more processors, cause the one or more processors to implement the crawler configuration method as described in the above embodiments.
[0017] According to one aspect of the embodiments of this application, a computer program product is provided, including a computer program, characterized in that, when the computer program is executed by a processor, it implements the crawler configuration method as described in the above embodiments.
[0018] In some embodiments of this application, the technical solutions provided enable users to quickly understand and select appropriate configuration modes through a visually intuitive interface, avoiding cumbersome text or command-line configurations. The corresponding configuration interface is displayed according to user needs, providing both simple and advanced templates, thus lowering the entry barrier for configuration. After the user's configuration operations directly form a preliminary solution, the debugging function allows users to verify the crawling effect in real time after configuration, quickly identifying potential problems. Real-time debugging provides real-time feedback, enabling users to promptly identify configuration errors, significantly reducing trial-and-error time during the crawling process and improving configuration accuracy and efficiency. This embodiment, by responding to each step of the user's operation and combining a visual interface with real-time debugging, significantly reduces the complexity of crawler configuration, allowing even non-professional users to easily and quickly configure crawler sources, improving user experience and configuration accuracy.
[0019] It should be understood that the above general description and the following detailed description are exemplary and explanatory only, and do not limit this application. Attached Figure Description
[0020] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application. It is obvious that the drawings described below are merely some embodiments of this application, and those skilled in the art can obtain other drawings based on these drawings without any inventive effort. In the drawings:
[0021] Figure 1 An exemplary implementation environment diagram is shown, in which the technical solutions of the embodiments of this application can be applied.
[0022] Figure 2 The diagram shows a flowchart of a web crawler configuration method provided in an embodiment of this application.
[0023] Figure 3 A schematic diagram of a simplified web crawler configuration interface provided in an embodiment of this application is shown.
[0024] Figure 4 A schematic diagram of an advanced web crawler configuration interface provided in an embodiment of this application is shown.
[0025] Figure 5 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S200 in the crawler configuration method shown in the corresponding embodiment.
[0026] Figure 6 It shows that according to Figure 5 A flowchart illustrating a specific implementation of step S230 in the crawler configuration method shown in the corresponding embodiment.
[0027] Figure 7 It shows that according to Figure 4 This is a schematic diagram of the crawling step configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0028] Figure 8 It shows that according to Figure 4 Another schematic diagram of the crawling step configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0029] Figure 9 It shows that according to Figure 4 This is a schematic diagram of the crawling step configuration area and the final result configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0030] Figure 10 It shows that according to Figure 4 This is a schematic diagram of the crawling step configuration area and the final result configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0031] Figure 11 It shows that according to Figure 4 This is a schematic diagram of the crawling step configuration area and the final result configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0032] Figure 12 It shows that according to Figure 4 This is a schematic diagram of the final result configuration area in the advanced crawler configuration interface shown in the corresponding embodiment.
[0033] Figure 13 It shows that according to Figure 2 A flowchart illustrating a specific implementation of step S300 in the crawler configuration method shown in the corresponding embodiment.
[0034] Figure 14 It shows that according to Figure 3 The simplified web crawler configuration interface and / or shown in the corresponding embodiment Figure 4 A schematic diagram of the single-step debugging interface in the advanced crawler configuration interface shown in the corresponding embodiment.
[0035] Figure 15 It shows that according to Figure 3 The simplified web crawler configuration interface and / or shown in the corresponding embodiment Figure 4 A schematic diagram of the single-step debugging interface in the advanced crawler configuration interface shown in the corresponding embodiment.
[0036] Figure 16 It shows that according to Figure 3 The simplified web crawler configuration interface and / or shown in the corresponding embodiment Figure 4 A schematic diagram of all debugging interfaces in the advanced crawler configuration interface shown in the corresponding embodiment.
[0037] Figure 17 A schematic diagram of a crawler configuration device provided in an embodiment of this application is shown.
[0038] Figure 18 A schematic diagram of the structure of a computer system for an electronic device provided in an embodiment of this application is shown. Detailed Implementation
[0039] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.
[0040] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.
[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.
[0042] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.
[0043] Figure 1 Here is a diagram illustrating the implementation environment of the crawler configuration method provided in one embodiment, such as... Figure 1 As shown, this implementation environment includes a web crawler configuration system 100. The web crawler configuration system 100 includes a system foreground 110 and a system background 120.
[0044] The system front-end 110 and system back-end 120 run within the same crawler configuration system 100, transmitting data to each other using corresponding instruction sets. The crawler configuration system 100 can run on various client devices.
[0045] The system front-end 110 interacts directly with the user, responding to user input and providing corresponding output. When the user selects a configuration mode through the system front-end 110, a selection command is sent to the system back-end 120. The system back-end 120 responds to this command, controlling the system front-end 110 to display the corresponding crawler configuration interface, which contains multiple crawler configuration items. The user configures each crawler configuration item through the system front-end 110's corresponding crawler configuration interface, and the system back-end 120 performs a preliminary configuration. When the user triggers the debugging function in the corresponding crawler configuration interface, the system back-end 120 executes the corresponding debugging steps according to the preliminary configuration and returns the corresponding debugging results. When the user confirms the debugging is complete through the system front-end 110, the system back-end 120 generates the final configuration.
[0046] It should be noted that the configuration scheme can be created not only on the user's client but also on the server side where the web crawler runs. The client and server can be devices such as smartphones, tablets, laptops, and desktop computers, but are not limited to these.
[0047] The client and server can connect via wired, wireless, or other communication methods, and this invention does not impose any restrictions on this.
[0048] The implementation details of the technical solutions in the embodiments of this application are described in detail below:
[0049] Figure 2 A flowchart of a crawler configuration method according to an embodiment of this application is shown. This crawler configuration method can be executed by a crawler configuration system, which can be... Figure 1 The web crawler configuration system 100 shown is illustrated. (Refer to...) Figure 2 As shown, this crawler configuration method includes at least the following:
[0050] S100, in response to the selection of the configuration mode, displays the corresponding crawler configuration interface, which contains multiple crawler configuration items.
[0051] S200, in response to the configuration operation of each of the crawler configuration items, a preliminary configuration scheme is obtained.
[0052] S300, in response to the triggering of the debugging function of the configuration interface, execute the corresponding debugging steps according to the preliminary configuration scheme.
[0053] S400, in response to the completion of debugging, forms the final configuration scheme.
[0054] In this embodiment, a visual interface allows users to quickly understand and select the appropriate configuration mode, avoiding cumbersome text or command-line configurations. The interface displays configuration options based on user needs, providing both simple and advanced templates, thus lowering the barrier to entry for configuration. After the user's configuration operations directly form a preliminary solution, the debugging function allows users to verify the crawling effect in real time after configuration, quickly identifying potential problems. Real-time debugging provides immediate feedback, enabling users to promptly identify configuration errors, significantly reducing trial-and-error time during the crawling process and improving configuration accuracy and efficiency. This embodiment, by responding to each step of user operations and combining a visual interface with real-time debugging, significantly reduces the complexity of crawler configuration, allowing even non-professional users to easily and quickly configure crawler sources, improving user experience and configuration accuracy.
[0055] In the S100, the visual crawler display interface is as follows: Figure 3 and Figure 4 As shown, the visual interface allows users to quickly understand and select the appropriate configuration mode, avoiding tedious text or command-line configuration. The interface displays configuration options tailored to user needs, providing both simple and advanced templates, thus lowering the barrier to entry for configuration.
[0056] Specifically, in some embodiments, the specific implementation of step S100 can be found in the following embodiments. This embodiment is based on... Figure 2According to the detailed description of step S100 in the crawler configuration method shown in the corresponding embodiment, step S100 in the crawler configuration method may include the following steps:
[0057] In response to the selection of the simplified mode, a simplified crawler configuration interface is displayed, which includes basic information configuration items, category page configuration items, results page configuration items, and result retrieval page configuration items.
[0058] In response to the selection of the advanced mode, an advanced crawler configuration interface is displayed, which includes basic configuration items, seed page configuration items, crawling step configuration items, final result configuration items, request configuration items, and additional parameter configuration items.
[0059] In this embodiment, by providing both simplified and advanced configuration interfaces, the different needs of various user groups in the web crawler configuration process are addressed. A highly adaptable configuration interface is provided for users with varying technical skill levels, improving the convenience and effectiveness of configuration.
[0060] Simple web crawler configuration interface as follows Figure 3 As shown, in the simplified mode, the simplified web crawler configuration interface provides intuitive and easy-to-understand configuration options for users with lower technical skills or simple needs, eliminating the need for them to understand complex technical details and lowering the entry barrier. Users can quickly complete the web crawler setup through configuration items such as basic information, category pages, and results pages, improving user efficiency.
[0061] Meanwhile, the basic information configuration items, category page configuration items, results page configuration items, and result retrieval page configuration items are all integrated into one page to reduce the time users spend switching between different pages, reduce the risk of missing information, and help users complete the configuration more smoothly.
[0062] Advanced web crawler configuration interface as follows Figure 4 As shown, in advanced mode, the advanced crawler configuration interface provides users with a certain technical background with more comprehensive and flexible configuration options, such as seed pages and crawling steps. This supports more complex crawling needs and allows users to fine-tune crawling behavior, enabling them to adjust settings according to specific crawling requirements. This improves the flexibility and accuracy of crawling, ensuring the crawler can effectively capture target data. The settings for basic configuration items, seed page configuration items, crawling step configuration items, final result configuration items, request configuration items, and additional parameter configuration items in the advanced crawler configuration interface meet the personalized configuration needs of advanced users. It also helps users better understand the role of each parameter, improving the transparency and controllability of the configuration.
[0063] In the S200, users can configure various crawler configuration items in the crawler configuration interface. After the user's configuration, a preliminary configuration plan is formed.
[0064] Specifically, in some embodiments, the specific implementation of step S200 can be found in the following embodiments. This embodiment is based on... Figure 2 In the detailed description of step S200 in the crawler configuration method shown in the corresponding embodiment, the crawler configuration interface is a simplified crawler configuration interface, and step S200 may include the following steps:
[0065] In response to configuration operations on each basic information configuration item in the basic information configuration area, basic information configuration information is generated.
[0066] In response to configuration operations on each category page configuration item in the category page configuration area, category page configuration information is generated.
[0067] In response to configuration operations on each configuration item in the result page configuration area, generate result page configuration information.
[0068] In response to configuration operations on each configuration item in the result page configuration area, result acquisition configuration information is generated.
[0069] The preliminary configuration scheme is formed based on the basic information configuration information, the category page configuration information, the result page configuration information, and the result acquisition configuration information.
[0070] In this embodiment, users can quickly complete the crawler setup by configuring basic information, category pages, results pages, and other configuration items in the simplified crawler configuration interface.
[0071] The basic information configuration items include configuration name, target source URL, whether to use a proxy, restart interval, whether to use rendering, category, and other basic information items. These are all necessary information items that users can intuitively input, which can avoid information omissions or errors and simplify the configuration process.
[0072] For target sources with multiple menu options within the entry URL, users can configure the selectors for the corresponding sub-menu elements through the category page configuration options to configure and crawl all sub-menus at once.
[0073] The results page configuration options help users accurately obtain target data, reducing uncertainty during the crawling process. It includes a results page sub-configuration option and a page number sub-configuration option. The results page configuration option configures the element selector for navigation to the details page within the list page, while the page number sub-configuration option configures the selector for pagination elements within the list page, allowing crawling of all pages.
[0074] The result retrieval configuration item is used to configure the content to be crawled in the details page. The content includes, but is not limited to, title, category, author, publication time, description, rating, download link, version number, etc., so as to filter the content crawled by the crawler, reduce the number of crawlers, and improve the accuracy of crawling.
[0075] Specifically, in some embodiments, the specific implementation of step S200 can be found in [reference needed]. Figure 5 . Figure 5 It is based on Figure 2 In the detailed description of step S200 in the crawler configuration method shown in the corresponding embodiment, the crawler configuration interface is an advanced crawler configuration interface, and step S200 may include the following steps:
[0076] S210, in response to configuration operations on each basic configuration item in the basic configuration area, obtains basic configuration information.
[0077] S220, in response to configuration operations on various subpage configuration items in the seed page configuration area, obtains seed page configuration information.
[0078] S230, in response to the configuration operation of each crawling step configuration item in the crawling step configuration area, obtain crawling action information.
[0079] S240, in response to the configuration operation of each final result configuration item in the final result configuration area, obtains the result configuration information.
[0080] S250, in response to the configuration operation on each requested configuration item in the requested configuration area, obtains the requested configuration information.
[0081] S260, in response to configuration operations on each additional parameter configuration item in the additional parameter configuration area, obtains additional parameter configuration information.
[0082] S270, the preliminary configuration scheme is formed based on the basic configuration information, the seed page configuration information, the crawling action information, the result configuration information, the request configuration information, and the additional parameter configuration information.
[0083] In this embodiment, the advanced crawler configuration interface modularizes the complex configuration process, enabling users to intuitively configure various settings. This reduces the learning cost while meeting the needs of professional users and improves the flexibility and adaptability of the crawler. Users can customize the crawler configuration according to specific needs to meet diverse application scenarios.
[0084] The basic configuration items in the basic configuration area include basic information about the crawler, such as configuration name, request delay, restart interval, and whether to use a proxy. These are all necessary information items that users can intuitively input, which can avoid information omissions or errors.
[0085] In the seed page configuration section, users can enter seed page information to determine the starting page for the crawler, ensuring the targeting and effectiveness of the crawl.
[0086] Users can configure the specific steps of the crawling process, including request methods and data parsing rules, in the crawling step configuration section. After the user completes the settings, crawling action information will be generated, enabling the crawler to crawl data according to the user-defined logic, improving efficiency and accuracy.
[0087] like Figure 12 As shown, in the final result configuration area, users can configure the processing method of the captured results, such as the storage format and output method. The final result configuration information can ensure that the captured results meet the user's needs and facilitate subsequent data analysis and utilization.
[0088] Within the request configuration area, users can configure more detailed request parameters, such as language, user agent (UA), referer, and content-type in the request headers. The resulting request configuration information can enhance the crawler's interaction with the target website and reduce the risk of being blocked.
[0089] In the additional parameter configuration area, users can configure additional parameters according to specific needs to obtain additional parameter configuration information, thereby improving the flexibility of the crawler and enabling it to adapt to diverse crawling scenarios.
[0090] Specifically, in some embodiments, the specific implementation of step S230 can be found in [reference needed]. Figure 6 . Figure 6 It is based on Figure 5 The crawler configuration method shown in the corresponding embodiment provides a detailed description of step S230. In this method, the crawling action information includes the execution order and crawling steps. Step S230 may include the following steps:
[0091] S232, in response to the sorting of configuration items for each crawling step, forms the execution order of the steps.
[0092] S234, in response to the configuration of sub-configuration items in the configuration items of each crawling step, forms a crawling step.
[0093] In this embodiment, by freely configuring the execution order and crawling steps, users can flexibly configure the crawler's crawling logic, thereby improving the crawler's operability and customizability.
[0094] In S232, the sorting of each crawling step configuration item can be flexibly defined. Specifically, each crawling step can be numbered. After the scheme is formed, the execution order of the steps can be formed according to the crawling step number, which allows users to flexibly configure the crawling logic and improves the operability and customizability of the crawler.
[0095] In S234, users can configure the sub-configuration items in the crawling step configuration items to enable the crawler to clearly execute the detailed operations of each step, such as request type and data parsing rules, thereby enhancing the targeting and effectiveness of crawling.
[0096] Specifically, in some embodiments, the specific implementation of step S234 can be found in the following embodiments. This embodiment is based on... Figure 6 According to the detailed description of step S234 in the crawler configuration method shown in the corresponding embodiment, step S234 in the crawler configuration method may include the following steps:
[0097] In response to the selection of a sub-configuration item for a crawling action in the crawling step configuration item, a sub-configuration item for a corresponding crawling action is formed, wherein the crawling action includes rendering crawling, Ajax crawling, JSON crawling, and ATOM / RSS crawling.
[0098] In response to the completion of the configuration of the sub-configuration item corresponding to the crawling action, the crawling steps corresponding to the crawling action are formed.
[0099] In this embodiment, users can flexibly select different types of crawling action sub-configuration items in the crawling step configuration items according to specific needs, realizing different crawling modes. This allows the crawling process to target different types of target sources and web page content, improving the flexibility, accuracy, and comprehensiveness of the crawling. The corresponding steps of the final crawling action are formed by flexibly configuring the crawler according to specific needs, and different selectors can be selected for data crawling based on different target source formats.
[0100] In this embodiment, each crawling step corresponds to a crawling type, categorizing different crawling actions to make the configuration process more systematic, reducing confusion and inconsistency, and lowering the risk of errors. Users can select the appropriate crawling method according to specific scenarios, enabling the crawler to effectively adapt to different crawling needs when faced with diverse web page content.
[0101] The crawling actions include different crawling modes such as rendering crawling, Ajax crawling, JSON crawling, and ATOM / RSS crawling, each targeting different sources.
[0102] Among them, such as Figure 7As shown, for target sources that require browser rendering to obtain dynamic content, rendering crawling mode can be used. Rendering crawling mode is used to handle web pages that require dynamic content. This type of web page content is usually loaded via JavaScript, while traditional crawlers can only obtain static HTML. Therefore, rendering crawling mode uses tools like Selenium and Playwright to simulate browser behavior to obtain the final rendered web page content.
[0103] In addition to basic rendering and fetching actions, the rendering and fetching mode also allows configuring simulated actions in sub-configuration items, such as clicking, continuous clicking of similar elements, target downloading, swiping, dropdown menu clicking, and refreshing. These can be specifically configured in the sub-configuration items. Additionally, you can configure the waiting time after a simulated action, configure the loop mode in the additional parameters, and enable dynamic click-based page turning.
[0104] Specifically, in the crawling step configuration area, each sub-configuration item contains at least one selection bar and at least one corresponding parameter bar. Users can select the corresponding options in the selection bar, such as operation type, DOM object to operate, wait time, etc.; and then enter the corresponding parameters in the parameter bar, such as click-normal, .click, 10ms, etc.
[0105] like Figure 8 As shown, the Ajax crawling mode is primarily used to handle content dynamically loaded via Ajax requests. Many modern websites use Ajax technology to request data in the background without reloading the page. The crawler needs to be configured with the correct request headers, request parameters, etc., to simulate these Ajax requests and thus obtain the data.
[0106] For requests to the target source that are made using the GET method by default, you can also make POST requests by configuring POST request parameters in Ajax fetch mode. At the same time, you can flexibly configure parameter items and parameter values.
[0107] like Figure 9 As shown, JSON data can be scraped using the JSON scraping mode. Many websites and applications provide RESTful APIs, which typically return data in JSON format. In a web crawler, you need to configure logic to parse the JSON data in order to extract the required information.
[0108] Specifically, after selecting the JSON selector, you can enter the corresponding configuration parameters. For example, links such as data.list.page. The configuration for data.list.page indicates that the links under the list objects within the data object will be retrieved and used as the URLs for the detail pages.
[0109] In the final result configuration area, you can also select the JSON selector and enter the corresponding configuration parameters, such as the title field, to indicate that while fetching the URL link, you should also access the details page and fetch the title field as the title.
[0110] like Figure 10 As shown, ATOM / RSS data formats can be scraped using the ATOM / RSS scraping mode. Content provided by the RSS or ATOM protocol is commonly used for publishing news, blog updates, etc. Web crawlers can obtain the latest content updates by parsing these XML-formatted data sources.
[0111] Specifically, after selecting the RSS selector or ATOM selector, you can enter the corresponding RSS link or ATOMlink link.
[0112] In the final results configuration area, you can also simultaneously select an RSS selector or an ATOM selector and configure the corresponding title to filter the crawled results.
[0113] In S240, the configuration in the final result configuration area is mainly for the details page. In addition to configuring JSON, RSS and other methods, it also supports CSS and XPath selectors, getting element attribute values, converting the main content to plain text (soup()), and extracting the download chain. It not only achieves the correspondence and unification with the crawling mode, but also realizes flexible configuration of the details page.
[0114] In some embodiments, S200 may further include:
[0115] In response to the configuration of regular expressions in the crawling step configuration area and / or the final result configuration area, regular expression matching rules are formed.
[0116] In this embodiment, as Figure 11 As shown, by adding regular expression configuration functionality to the crawling step configuration area and the final result configuration area, all crawling modes can support regular expression matching. By configuring regular expressions, secondary processing can be performed to obtain more accurate data.
[0117] In the S300, after the user's configuration operations directly form a preliminary solution, the debugging function allows users to verify the capture effect in real time after configuration is completed, quickly identifying potential problems. Real-time debugging provides real-time feedback, enabling users to promptly identify configuration errors, significantly reducing the trial-and-error time during the capture process and improving the accuracy and efficiency of configuration.
[0118] Specifically, in some embodiments, the specific implementation of step S300 can be found in [reference needed]. Figure 13 . Figure 13 It is based on Figure 2According to the detailed description of step S300 in the crawler configuration method shown in the corresponding embodiment, step S300 in the crawler configuration method may include the following steps:
[0119] S310, in response to the selection of the debug menu of the configuration interface, switch to the corresponding debug interface, the debug interface including a single-step debug interface and a full debug interface.
[0120] S320, in response to the triggering of the debugging function in the debugging interface, execute the corresponding debugging steps according to the preliminary configuration scheme.
[0121] In this embodiment, the debugging function includes single-step debugging and full debugging. Different debugging modes provide flexibility, allowing users to choose the most suitable debugging method according to their actual needs, greatly improving the efficiency and accuracy of debugging. Single-step debugging allows for gradual observation of the configuration effect, which is suitable for locating specific problems; while full debugging allows all steps to be run at once, which is suitable for overall verification.
[0122] In the S310, the debug menu is integrated into the crawler configuration interface, whether it's the simple crawler configuration interface or the advanced crawler configuration interface. The debug menu is placed side by side with various crawler configuration items on the same screen. Users can not only switch between single-step debugging and full debugging through the touch debug menu, but also perform real-time output debugging in conjunction with the crawler configuration items on the side. This allows users to adjust and optimize the configuration more flexibly, reducing debugging time and complexity.
[0123] In the S320, by responding instantly to user actions, the system can automatically perform debugging based on the configuration, eliminating the need for manual checks and reducing the possibility of human error. This allows for faster identification and resolution of configuration issues, improving overall data capture efficiency.
[0124] Specifically, in some embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 13 According to the detailed description of step S320 in the crawler configuration method shown in the corresponding embodiment, step S320 in the crawler configuration method may include the following steps:
[0125] In response to the triggering of the debugging function corresponding to each capture step in the debugging interface, the corresponding capture step is executed.
[0126] The output information corresponding to each crawling step is displayed in the single-step debugging interface.
[0127] In this embodiment, as Figure 14 and Figure 15As shown, single-step debugging focuses on each specific crawling step, rather than processing information from all steps at once. In the single-step debugging interface, users can view debugging information for each crawling step, clearly understanding the status and execution results of each step, which helps in quickly identifying problems. After the user triggers the debugging function in the debugging interface, the system executes the corresponding crawling steps according to the initial configuration plan. That is, during the debugging process, users can check the configuration effect step by step according to actual needs without worrying about affecting subsequent steps. By executing step by step and displaying the results of each step in real time, users can immediately see the execution effect and data feedback. This helps users quickly determine the effectiveness and accuracy of the steps, effectively locate errors, and thus adjust the configuration plan in a timely manner, further improving the success rate of crawling.
[0128] In single-step debugging mode, you can freely select the source URL, submenu URL, list page URL, and details page URL, and then display the final results. The refresh button ensures that the crawled results are updated while maintaining the current step.
[0129] Specifically, in some embodiments, the specific implementation of step S320 can be found in the following embodiments. This embodiment is based on... Figure 13 According to the detailed description of step S320 in the crawler configuration method shown in the corresponding embodiment, step S320 in the crawler configuration method may include the following steps:
[0130] In response to the triggering of the debugging functions in all the debugging interfaces, the preliminary configuration scheme is executed.
[0131] The output results corresponding to the preliminary configuration scheme are displayed in the entire debugging interface.
[0132] In this embodiment, as Figure 16 As shown, the entire debugging focuses on the overall configuration scheme. In the entire debugging interface, the system will start from the entry URL, randomly flip through pages, randomly select detail pages, and display the crawling results of the detail pages. Users can fully understand the crawling process and avoid missing key information.
[0133] In S400, after debugging, a final configuration scheme is generated. Based on this configuration scheme, a corresponding crawler can be generated to crawl network resources.
[0134] The following describes an embodiment of the apparatus described in this application, which can be used to execute the web crawler configuration method described in the above embodiments of this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the web crawler configuration method described above.
[0135] Figure 17 A block diagram of a crawler configuration apparatus according to an embodiment of this application is shown.
[0136] Reference Figure 17 As shown, a crawler configuration device 700 according to an embodiment of this application is applied to a crawler configuration system including a visual crawler configuration interface. The crawler configuration device includes: an interface display module 710, used to display a corresponding crawler configuration interface in response to the selection of a configuration mode, the crawler configuration interface including multiple crawler configuration items; a preliminary configuration module 720, used to obtain a preliminary configuration scheme in response to the configuration operation of each of the crawler configuration items; a step debugging module 730, used to execute corresponding debugging steps according to the preliminary configuration scheme in response to the triggering of the debugging function of the configuration interface; and a scheme formation module 740, used to form a final configuration scheme in response to the completion of debugging.
[0137] In some feasible embodiments of this application, the configuration mode includes a simple mode and an advanced mode. The interface display module 710 specifically includes: a simple interface submodule, used to display a simple crawler configuration interface in response to the selection of the simple mode, the simple crawler configuration interface including basic information configuration items, category page configuration items, result page configuration items, and result acquisition page configuration items; and an advanced interface submodule, used to display an advanced crawler configuration interface in response to the selection of the advanced mode, the advanced crawler configuration interface including basic configuration items, seed page configuration items, crawling step configuration items, final result configuration items, request configuration items, and additional parameter configuration items.
[0138] In some feasible embodiments of this application, the crawler configuration interface is a simplified crawler configuration interface. The preliminary configuration module 720 specifically includes: a basic information configuration submodule, used to generate basic information configuration information in response to configuration operations on each basic information configuration item in the basic information configuration area; a category page configuration submodule, used to generate category page configuration information in response to configuration operations on each category page configuration item in the category page configuration area; a result page configuration submodule, used to generate result page configuration information in response to configuration operations on each result page configuration item in the result page configuration area; a result acquisition page submodule, used to generate result acquisition configuration information in response to configuration operations on each result acquisition page configuration item in the result acquisition page configuration area; and a first generation submodule, used to form the preliminary configuration scheme based on the basic information configuration information, the category page configuration information, the result page configuration information, and the result acquisition configuration information.
[0139] In some feasible embodiments of this application, the crawler configuration interface is an advanced crawler configuration interface. The preliminary configuration module 720 specifically includes: a basic configuration submodule, used to obtain basic configuration information in response to configuration operations on each basic configuration item in the basic configuration area; a seed page configuration submodule, used to obtain seed page configuration information in response to configuration operations on various subpage configuration items in the seed page configuration area; a crawling step configuration submodule, used to obtain crawling action information in response to configuration operations on each crawling step configuration item in the crawling step configuration area; and a final result configuration submodule, used to obtain crawling action information in response to configuration operations on the final result. The configuration operations of each final result configuration item in the final result configuration area yield result configuration information; the request configuration submodule is used to respond to the configuration operations of each request configuration item in the request configuration area to obtain request configuration information; the additional parameter configuration submodule is used to respond to the configuration operations of each additional parameter configuration item in the additional parameter configuration area to obtain additional parameter configuration information; the second generation submodule is used to form the preliminary configuration scheme based on the basic configuration information, the seed page configuration information, the crawling action information, the result configuration information, the request configuration information, and the additional parameter configuration information.
[0140] In some feasible embodiments of this application, the crawling action information includes the step execution order and crawling steps; the crawling step configuration submodule specifically includes: a step sorting unit, used to form a step execution order in response to the sorting of each crawling step configuration item; and a step configuration unit, used to form crawling steps in response to the configuration of sub-configuration items in each crawling step configuration item.
[0141] In some feasible embodiments of this application, the step configuration unit is specifically used to perform: in response to the selection of a sub-configuration item for a crawling action in the crawling step configuration item, forming a sub-configuration item for a corresponding crawling action, wherein the crawling action includes rendering crawling, Ajax crawling, JSON crawling, and ATOM / RSS crawling; in response to the completion of the configuration of the sub-configuration item corresponding to the crawling action, forming a crawling step corresponding to the crawling action.
[0142] In some feasible embodiments of this application, the preliminary configuration module 720 further includes: a regular expression configuration unit, used to form regular expression matching rules in response to the configuration of regular expressions in the crawling step configuration area and / or the final result configuration area.
[0143] In some feasible embodiments of this application, the step debugging module 730 specifically includes: a mode selection submodule, used to switch to the corresponding debugging interface in response to the selection of the debugging menu of the configuration interface, the debugging interface including a single-step debugging interface and a full debugging interface; and a function triggering submodule, used to execute the corresponding debugging steps according to the preliminary configuration scheme in response to the triggering of the debugging function in the debugging interface.
[0144] In some feasible embodiments of this application, the debugging interface is a single-step debugging interface, which displays the debugging information of each crawling step in the preliminary configuration scheme; the function triggering submodule specifically includes: a single-step execution unit, used to execute the corresponding crawling step in response to the triggering of the debugging function corresponding to each crawling step in the debugging interface; and a single-step result unit, used to display the output information corresponding to each crawling step in the single-step debugging interface.
[0145] In some feasible embodiments of this application, the debugging interface is a full debugging interface; the function triggering submodule specifically includes: a full execution unit, used to execute the preliminary configuration scheme in response to the triggering of the debugging function in the full debugging interface; and a full result unit, used to display the output results corresponding to the preliminary configuration scheme in the full debugging interface.
[0146] In this embodiment, a visual interface allows users to quickly understand and select the appropriate configuration mode, avoiding cumbersome text or command-line configurations. The interface displays configuration options based on user needs, providing both simple and advanced templates, thus lowering the barrier to entry for configuration. After the user's configuration operations directly form a preliminary solution, the debugging function allows users to verify the crawling effect in real time after configuration, quickly identifying potential problems. Real-time debugging provides immediate feedback, enabling users to promptly identify configuration errors, significantly reducing trial-and-error time during the crawling process and improving configuration accuracy and efficiency. This embodiment, by responding to each step of user operations and combining a visual interface with real-time debugging, significantly reduces the complexity of crawler configuration, allowing even non-professional users to easily and quickly configure crawler sources, improving user experience and configuration accuracy.
[0147] Figure 18 A schematic diagram of the structure of a computer system suitable for implementing the electronic device of the present application is shown.
[0148] It should be noted that, Figure 18 The computer system of the electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0149] like Figure 18As shown, the computer system includes a Central Processing Unit (CPU) 1801, which can perform various appropriate actions and processes based on programs stored in Read-Only Memory (ROM) 1802 or programs loaded from storage portion 1808 into Random Access Memory (RAM) 1803, such as performing the methods described in the above embodiments. The RAM 1803 also stores various programs and data required for system operation. The CPU 1801, ROM 1802, and RAM 1803 are interconnected via a bus 1804. An Input / Output (I / O) interface 1805 is also connected to the bus 1804.
[0150] The following components are connected to I / O interface 1805: an input section 1806 including a keyboard, mouse, etc.; an output section 1807 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.; a storage section 1808 including a hard disk, etc.; and a communication section 1809 including a network interface card such as a LAN (Local Area Network) card, modem, etc. The communication section 1809 performs communication processing via a network such as the Internet. A drive 1810 is also connected to I / O interface 1805 as needed. Removable media 1811, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on drive 1810 as needed so that computer programs read from them can be installed into storage section 1808 as needed.
[0151] Specifically, according to embodiments of this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program including a computer program for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication section 1809, and / or installed from removable medium 1811. When the computer program is executed by central processing unit (CPU) 1801, it performs various functions defined in the system of this application.
[0152] It should be noted that the computer-readable medium shown in the embodiments of this application can be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), flash memory, optical fiber, portable compact disc read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying a computer-readable computer program. The transmitted data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The computer program contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.
[0153] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. Each block in a flowchart or block diagram may represent a module, segment, or portion of code, which contains one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0154] The units described in the embodiments of this application can be implemented in software or hardware, and the described units can also be located in a processor. The names of these units do not necessarily limit the specific unit itself.
[0155] In another aspect, this application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiments; or it may exist independently and not assembled into the electronic device. The computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods described in the above embodiments.
[0156] This specification also provides a computer program product that stores at least one instruction, said at least one instruction being loaded and executed by the processor as described above. Figures 1 to 16 The method described in the illustrated embodiment can be found in the following document for a detailed execution process. Figures 1 to 16 The specific details of the illustrated embodiments will not be elaborated here.
[0157] It should be noted that although several modules or units for the device used to perform actions have been mentioned in the detailed description above, this division is not mandatory. In fact, according to the embodiments of this application, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided and embodied by multiple modules or units.
[0158] Through the above description of the embodiments, those skilled in the art will readily understand that the exemplary embodiments described herein can be implemented by software or by combining software with necessary hardware. Therefore, the technical solutions according to the embodiments of this application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (such as a CD-ROM, USB flash drive, external hard drive, etc.) or on a network, including several instructions to cause a computing device (such as a personal computer, server, touch terminal, or network device, etc.) to execute the method according to the embodiments of this application.
[0159] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.
[0160] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.
Claims
1. A web crawler configuration method, characterized in that, The crawler configuration method, applied to a crawler configuration system that includes a visual crawler configuration interface, comprises: In response to the selection of the configuration mode, the corresponding crawler configuration interface is displayed, which contains multiple crawler configuration items. In response to the configuration operations on each of the crawler configuration items, a preliminary configuration scheme is obtained; In response to the triggering of the debugging function of the configuration interface, the corresponding debugging steps are executed according to the preliminary configuration scheme; Upon completion of debugging, the final configuration scheme is formed.
2. The crawler configuration method as described in claim 1, characterized in that, The configuration modes include a simple mode and an advanced mode. In response to the selection of a configuration mode, a corresponding crawler configuration interface is displayed, specifically including: In response to the selection of the simplified mode, a simplified crawler configuration interface is displayed, which includes basic information configuration items, category page configuration items, result page configuration items, and result retrieval page configuration items. In response to the selection of the advanced mode, an advanced crawler configuration interface is displayed, which includes basic configuration items, seed page configuration items, crawling step configuration items, final result configuration items, request configuration items, and additional parameter configuration items.
3. The crawler configuration method as described in claim 2, characterized in that, The crawler configuration interface is a simplified crawler configuration interface. The initial configuration scheme is obtained in response to configuration operations on each of the crawler configuration items, specifically including: In response to configuration operations on each basic information configuration item in the basic information configuration area, basic information configuration information is generated; In response to configuration operations on each category page configuration item in the category page configuration area, generate category page configuration information; In response to configuration operations on each configuration item in the result page configuration area, generate result page configuration information; In response to configuration operations on each configuration item in the configuration area of the result page, generate result acquisition configuration information; The preliminary configuration scheme is formed based on the basic information configuration information, the category page configuration information, the result page configuration information, and the result acquisition configuration information.
4. The crawler configuration method as described in claim 2, characterized in that, The crawler configuration interface is an advanced crawler configuration interface. The initial configuration scheme is obtained in response to configuration operations on each of the crawler configuration items, specifically including: In response to configuration operations on each basic configuration item in the basic configuration area, basic configuration information is obtained; In response to configuration operations on various subpage configuration items in the seed page configuration area, the seed page configuration information is obtained; In response to configuration operations on each crawling step configuration item in the crawling step configuration area, crawling action information is obtained; In response to configuration operations on each final result configuration item in the final result configuration area, result configuration information is obtained; In response to configuration operations on each request configuration item in the request configuration area, request configuration information is obtained; In response to configuration operations on each additional parameter configuration item in the additional parameter configuration area, additional parameter configuration information is obtained; The preliminary configuration scheme is formed based on the basic configuration information, the seed page configuration information, the crawling action information, the result configuration information, the request configuration information, and the additional parameter configuration information.
5. The crawler configuration method as described in claim 4, characterized in that, The grabbing action information includes the execution order of steps and the grabbing steps; the grabbing action information obtained in response to the configuration operation of each grabbing step configuration item in the grabbing step configuration area specifically includes: The execution order of the steps is determined by sorting the configuration items for each crawling step. A crawling step is formed in response to the configuration of sub-configuration items in the configuration items of each crawling step.
6. The crawler configuration method as described in claim 5, characterized in that, The process of forming a crawling step in response to the configuration of sub-configuration items in each step configuration item specifically includes: In response to the selection of a sub-configuration item for a crawling action in the crawling step configuration item, a sub-configuration item for a corresponding crawling action is formed, wherein the crawling action includes rendering crawling, Ajax crawling, JSON crawling, and ATOM / RSS crawling; In response to the completion of the configuration of the sub-configuration item corresponding to the crawling action, the crawling steps corresponding to the crawling action are formed.
7. A crawler configuration device, characterized in that, The crawler configuration device, applicable to a crawler configuration system that includes a visual crawler configuration interface, comprises: The interface display module is used to display the corresponding crawler configuration interface in response to the selection of the configuration mode. The crawler configuration interface contains multiple crawler configuration items. The preliminary configuration module is used to obtain a preliminary configuration scheme in response to the configuration operations of each of the crawler configuration items; The step debugging module is used to respond to the triggering of the debugging function of the configuration interface and execute the corresponding debugging steps according to the preliminary configuration scheme. The scheme formation module is used to generate the final configuration scheme in response to the completion of debugging.
8. A computer-readable medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the crawler configuration method as described in any one of claims 1 to 6.
9. An electronic device, characterized in that, include: One or more processors; A storage device for storing one or more programs, which, when executed by one or more processors, cause the one or more processors to implement the crawler configuration method as described in any one of claims 1 to 6.
10. A computer program product comprising one or more computer programs, characterized in that, When the one or more computer programs are executed by one or more processors, they implement the steps of the crawler configuration method according to any one of claims 1 to 6.