Data crawling methods, systems, and computer-readable storage media

Abstract

This application provides a data crawling method, system, and computer-readable storage medium. The method is applied to a data crawling system, which includes a web client and a server. The method includes: the web client displaying a first page for configuring a data crawling task; the web client receiving a first configuration operation for the data crawling task through the first page, and sending the configuration content for the data crawling task to the server for storage. The first configuration operation includes at least configuring the data source and storage medium for the data crawling task, and the storage medium is used to store crawled web page data; when the web client receives an operation to start the data crawling task, it sends a task request to the server, requesting the server to execute the data crawling task; the server crawls web page data from the URL corresponding to the data source, and saves the web page data to the configured storage medium. This improves the efficiency of data crawling.

Description

Technical Field

[0001] This application relates to the field of data acquisition technology, specifically to a data crawling method, system, and computer-readable storage medium. Background Technology

[0002] With the rapid development of information network technology, the amount of network information has exploded. However, there is usually some redundant information or information that users do not care about in the huge amount of network information. Therefore, it is necessary to extract the information that users need from the huge amount of network information.

[0003] A web crawler (or crawler engine) is a program that automatically browses the web and retrieves web page data. It can filter and retrieve the information a user needs from the web based on certain search strategies. Because web page data is complex and its content varies greatly, sometimes only a small portion of the data needs to be retrieved. Therefore, developers need to write a lot of code to complete the web crawling process and save the retrieved data to the storage medium specified in the code after completion.

[0004] However, if business needs require changing or adding storage media to save the data crawling results, the code needs to be modified to achieve this. This method of writing code consumes a lot of manpower and the data crawling efficiency is relatively low. Summary of the Invention

[0005] This application provides a data crawling method, system, and computer-readable storage medium that can improve the efficiency of data crawling.

[0006] In a first aspect, this application provides a data crawling method applied to a data crawling system, the data crawling system including a web client and a server, the method comprising:

[0007] The web client displays a first page, which is used to configure the data crawling task;

[0008] The web client receives a first configuration operation for the data crawling task through the first page, and sends the configuration content of the data crawling task to the server for storage. The first configuration operation includes at least the configuration of the data crawling data source and storage medium of the data crawling task, and the storage medium is used to store the crawled web page data.

[0009] When the web client receives an operation to start the data crawling task, it sends a task request to the server, requesting the server to execute the data crawling task;

[0010] The server crawls web page data from the URL corresponding to the crawling data source and saves the web page data to the configured storage medium.

[0011] The web client and server can communicate with each other. Users (such as developers) can log in to the web management page on the web client and configure data crawling tasks (i.e., create data crawling tasks) on the web management page. After triggering the start of the data crawling task on the web client, the web client can send the task request to the server, which can then execute the data crawling task in the background and store the data crawling results.

[0012] When configuring data crawling tasks on the web client, the web client can display a first page (i.e., the data crawling task management page). On this page, users can configure information such as the data source and storage medium for the data crawling task, and can also send the configuration content to the server for storage. For example, the server stores the configuration content of the data crawling task in a first data table. During the data crawling process on the server, web page data can be crawled from the corresponding URLs according to the configured data source. Furthermore, when configuring the storage medium for the data crawling task on the web client, one or more storage media can be configured according to business needs. Then, after the server crawls the web page data, it can store the web page data in one or more configured storage media. If the user needs to change or add storage media, they only need to re-select the configuration on the first page of the web client; no further code changes are required.

[0013] Therefore, the above data crawling method, by dynamically configuring the storage medium for storing the data crawling results on the web client, can reduce the human cost of writing code. At the same time, the dynamic configuration method has high flexibility, can meet the needs of different business applications, and improves the efficiency of data crawling.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the first page includes a first control, and the aforementioned web client receives a first configuration operation for the data crawling task through the first page, including:

[0015] In response to the user's click on the first control, the web client displays a first input control for configuring the data crawling task;

[0016] The system receives the crawling data source and the storage medium input by the user on the first input control, and completes the first configuration operation.

[0017] The first page may include a first control (i.e., a new control). After the user clicks this first control, the web client can display a first input control for configuring the data crawling task. This input control can be used to enter information such as name, code, description, data source, and storage medium. After entering these configuration details, saving the settings completes the first configuration operation. Thus, users can easily configure the data crawling task through a convenient web page configuration process, improving the efficiency of data crawling.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, the way a user inputs the storage medium on the first input control includes: selecting at least one storage medium from a list of created storage media.

[0019] In other words, when configuring the above storage media, multiple storage media can be created in advance, forming a storage media list. Then, the user can select at least one storage media from the storage media list as a configuration item for the above data crawling task, thereby reducing the cumbersomeness of the configuration process and further improving the efficiency of data crawling.

[0020] In conjunction with the first aspect, in some implementations of the first aspect, the above method also includes:

[0021] The web client displays a second page, which is used to configure the storage medium;

[0022] The web client receives a second configuration operation on the storage medium through the second page, and sends the configuration content of the storage medium to the server storage. The second configuration operation includes at least connection configuration of the storage medium.

[0023] Since users can configure the storage medium for data crawling tasks on the first page of the web client, it's necessary to configure the storage medium in advance for later selection. When configuring the storage medium on the web client, the web client can display a second page (the storage medium configuration page). On this page, users can configure information such as the connection settings for the storage medium and send the configuration content to the server for storage. For example, the server stores the connection configuration information for the storage medium in a fifth data table.

[0024] This allows users to dynamically configure storage media on the web client, reducing the human cost of writing code and improving the efficiency of data retrieval.

[0025] In conjunction with the first aspect, in some implementations of the first aspect, the second page includes a second control, and the web client receives a second configuration operation on the storage medium through the second page, including:

[0026] In response to the user's click on the second control, the web client displays a third page for configuring the connection to the storage medium;

[0027] Receive the storage source selection operation input by the user on the third page, and display a second input control configured according to the selected storage source;

[0028] The system receives connection configuration information for the storage medium input by the user on the second input control and completes the second configuration operation. The connection configuration information includes information on creating data tables or indexes based on the selected storage source.

[0029] The second page may include a second control (a "clickable configuration" control). After the user clicks this second control, the web client can display a third page (the storage connection configuration page). This third page displays a storage source selection control, allowing the user to input their storage source choice. If the user selects a storage source, the third page will then display a corresponding second input control (a configuration input box) for inputting the connection configuration information for that storage medium. For example, if the storage source is selected to search MySQL, the connection configuration information may include information about creating data tables; if the storage source is selected to search Kafka, the connection configuration information may include information about creating topics; and if the storage source is selected to search Elasticsearch, the connection configuration information may include information about creating indexes. This allows for the configuration of the storage medium, enabling subsequent data crawling tasks to select the appropriate storage medium.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the above method also includes:

[0031] The web client displays a fourth page, which is used to configure the storage source. The storage source is selected by the user when the user enters the storage source selection operation on the third page.

[0032] The third configuration operation for the storage source is received through the fourth page, and the third configuration operation includes at least the configuration of the hostname of the storage source.

[0033] Since the storage source was selected during the storage media configuration, it also needs to be configured beforehand. Accordingly, the web client can display a fourth page (the storage source management page). This page can include an add control. After the user clicks the add control, the fourth page displays the information required for configuring the storage source, such as name, hostname, username, password, and port. After the user fills in this information (i.e., enters the third configuration step), the configuration of a storage source is complete.

[0034] In conjunction with the first aspect, in some implementations of the first aspect, after the web client receives the first configuration operation for the data crawling task through the first page, the method further includes:

[0035] The web client displays the configuration content of the data crawling task and a third control on the first page. The third control is used to trigger the start and stop of the data crawling task.

[0036] After configuring the data crawling task, the first page of the web client will display the configured data crawling task record. This record will have a corresponding third control (i.e., operation control) to trigger the start and stop of the task, and can also be used to trigger the editing of the data crawling task, i.e., to change the configuration content such as storage medium.

[0037] Accordingly, the operation of the aforementioned Web client to start the data crawling task includes: the Web client receiving the user's click operation on the third control, triggering the start of the data crawling task.

[0038] As a result, users can trigger and start data crawling tasks with a single click in the web client, reducing the human cost of writing code and improving efficiency.

[0039] In conjunction with the first aspect, in some implementations of the first aspect, saving the webpage data to the configured storage medium includes:

[0040] The server obtains the storage medium corresponding to the data crawling task based on the identifier of the data crawling task;

[0041] If the information of the storage medium is not empty, the server obtains the connection configuration information corresponding to the storage medium;

[0042] Based on the connection configuration information, the webpage data is saved to the storage medium.

[0043] Because the web client is configured with the storage medium corresponding to the data crawling task, and the server stores the configuration content corresponding to the data crawling task, the server can store the crawled web page data in the corresponding storage medium after completing the data crawling process. Here, the server obtains the connection configuration information of the storage medium, such as the created data tables or indexes, and then saves the web page data to the corresponding storage medium according to this connection configuration information. Therefore, the server can store web page data according to the configured storage medium, and can dynamically adjust the save location when the configured storage medium changes, thereby improving data crawling efficiency.

[0044] In conjunction with the first aspect, in some implementations of the first aspect, saving the webpage data to the storage medium according to the connection configuration information includes:

[0045] The server determines the type of the storage medium;

[0046] If the connection configuration information is not empty, a connection client corresponding to the storage medium is created according to the type of the storage medium and the connection configuration information;

[0047] According to the connection client, the web page data is saved to the storage medium.

[0048] As described above, a web client can be configured with one or more storage media, meaning it can include different types of storage media. It's understood that different types of storage media correspond to different connection configuration information. Therefore, when storing web page data, the server can first determine the type of the configured storage media, and then, based on that type and connection configuration information, create a connection client corresponding to that storage media to establish a connection. Subsequently, the web page data can be saved to the corresponding storage media based on this connection client.

[0049] In conjunction with the first aspect, in some implementations of the first aspect, after creating the connection client corresponding to the storage medium, the method further includes:

[0050] The server stores information about the connected client.

[0051] In other words, once a client connection is successfully established, the server can save the client's information to facilitate continuous data storage. This information can then be directly retrieved for webpage data storage, further improving the efficiency of data crawling.

[0052] In conjunction with the first aspect, in some implementations of the first aspect, the storage medium includes at least one of the following: MySQL type storage medium, Kafka type storage medium, and Elasticsearch type storage medium.

[0053] That is, the storage medium configured above can be at least one of the following: MySQL type storage medium, Kafka type storage medium, and Elasticsearch type storage medium. Of course, this application is not limited to these three types and can include more or fewer types of storage media according to actual needs.

[0054] Secondly, this application provides a data crawling system, which includes a web client and a server, and is used to execute any one of the methods in the technical solution of the first aspect above.

[0055] Thirdly, this application provides an apparatus included in a data crawling system, which has the functions of implementing the first aspect and its possible implementations. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules or units corresponding to the above functions. For example, a receiving module or unit, a processing module or unit, etc.

[0056] Fourthly, this application provides a web client, including: one or more processors, and memory;

[0057] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, which the one or more processors call to cause the web client to execute the corresponding process in the first aspect of the technical solution.

[0058] Fifthly, this application provides a server, including: one or more processors, and memory;

[0059] The memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code including computer instructions, which the one or more processors call to cause the server to perform the corresponding process in the first aspect of the technical solution.

[0060] In a sixth aspect, this application provides a computer-readable storage medium including instructions that, when executed on a data crawling system, cause the data crawling system to perform any one of the methods in the first aspect of the technical solution.

[0061] In a seventh aspect, this application provides a computer program product, which includes computer program code. When the computer program code is run on a data crawling system, it causes the data crawling system to execute any one of the methods in the technical solution of the first aspect. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the system architecture of a data crawling method provided in an embodiment of this application;

[0063] Figure 2 This is a schematic diagram of a web management page on a web client provided in an embodiment of this application;

[0064] Figure 3 This is a schematic diagram of a data crawling task management page on a web client provided in an embodiment of this application;

[0065] Figure 4 This is another example of a data crawling task management page on a web client provided in this application embodiment;

[0066] Figure 5 This is another example of a data crawling task management page on a web client provided in the embodiments of this application;

[0067] Figure 6 This is another example of a data crawling task management page on a web client provided in the embodiments of this application;

[0068] Figure 7 This is another example of a data crawling task management page on a web client provided in the embodiments of this application;

[0069] Figure 8 This is a schematic diagram of a URL management page on a web client provided in an embodiment of this application;

[0070] Figure 9 This is another example of a URL management page on a web client provided in this application embodiment;

[0071] Figure 10 This is another example of a URL management page on a web client provided in the embodiments of this application;

[0072] Figure 11 This is another example of a URL management page on a web client provided in the embodiments of this application;

[0073] Figure 12 This is a schematic diagram of a storage source management page on a web client provided in an embodiment of this application;

[0074] Figure 13This is a schematic diagram of a storage media configuration page on a web client provided in an embodiment of this application;

[0075] Figure 14 This is another example of a storage media configuration page on a web client provided in this application embodiment;

[0076] Figure 15 This is another example of a storage media configuration page on a web client provided in the embodiments of this application;

[0077] Figure 16 This is another example of a storage media configuration page on a web client provided in the embodiments of this application;

[0078] Figure 17 This is another example of a storage media configuration page on a web client provided in the embodiments of this application;

[0079] Figure 18 This is another example of a storage media configuration page on a web client provided in the embodiments of this application;

[0080] Figure 19 This is another example of a storage media configuration page on a web client provided in the embodiments of this application;

[0081] Figure 20 This is a flowchart illustrating an example of a data crawling method provided in an embodiment of this application;

[0082] Figure 21 This is a flowchart illustrating another data crawling method provided in an embodiment of this application;

[0083] Figure 22 This is a flowchart illustrating yet another data crawling method provided in an embodiment of this application;

[0084] Figure 23 This is a schematic diagram of the structure of an example Web client provided in an embodiment of this application;

[0085] Figure 24 This is a schematic diagram of the structure of an example server provided in an embodiment of this application;

[0086] Figure 25 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0087] The technical solutions of the embodiments of this application will be described below with reference to the accompanying drawings. In the description of the embodiments of this application, unless otherwise stated, " / " means "or," for example, A / B can mean A or B; "and / or" in this text is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Furthermore, in the description of the embodiments of this application, "multiple" refers to two or more than two.

[0088] Hereinafter, the terms "first," "second," and "third" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first," "second," or "third" may explicitly or implicitly include one or more of that feature.

[0089] In today's environment of explosive growth in online information, it is difficult for users to obtain the information they need from the vast amount of online data. This has led to the development of the concept of search engines. A search engine serves as a gateway for users to access the World Wide Web, assisting them in finding the information they require. However, considering that users from different fields and backgrounds often have different search objectives and needs, the results returned by search engines may also include web page content that users are not interested in. To solve this problem, web crawlers, which target and extract web page data, have emerged. They can automatically download web page data and then filter and extract the information needed by users according to certain search strategies, and have a wide range of applications.

[0090] During data crawling, webpage data is often complex and varied, sometimes requiring only a small portion of the data. For example, when crawling trending topics with high user interest, only the title, author, and popularity score need to be retrieved, without other data. Therefore, developers need to write code to align with the regular expressions used in data crawling, and modify the code to crawl different webpage data depending on user needs. Furthermore, after crawling the webpage data, the code logic can be used to save the results to a fixed storage medium. This is typically set in the code; if the storage medium needs to be changed or added due to business requirements, the code also needs to be modified. Clearly, this coding approach is labor-intensive, and the method of changing storage media is inflexible, resulting in low data crawling efficiency.

[0091] In view of this, the present application provides a data crawling method that allows developers to dynamically configure the storage medium for saving data crawling results on a web management page, reducing the manpower cost of writing code. The dynamic configuration method has high flexibility and improves the efficiency of data crawling.

[0092] Below, we will first introduce the application scenarios of the data crawling method in this application. In some embodiments, this data crawling method can be applied to, for example... Figure 1 The system architecture shown can include a web client, a server, and electronic devices. The web client can communicate with the server, and the electronic devices can also communicate with the server.

[0093] The system allows developers to log in and access a web management page via a web client. This page provides crawler management functionality, including data crawling task management, Uniform Resource Locator (URL) management, and storage media management. The data crawling task management function allows users to create and control the start and stop of crawling tasks, set crawling intervals for scheduled crawling, and configure the storage media for saving crawled results. The URL management function allows users to create and configure URL data sources. These URLs can be selected when creating a crawling task to determine which data source to crawl (i.e., which URL to crawl). The storage media management function allows users to create and configure storage media. These media can be selected when creating a crawling task to determine where the crawled results will be saved. After a data crawling task is initiated on the web client, the web client can send the task request to the server. The server can then execute the data crawling task in the background and save the crawled results to the configured storage medium. In some implementations, the data crawling task can be executed continuously, constantly updating the stored data crawling results.

[0094] For the aforementioned electronic devices (e.g., mobile phones), if a business application corresponding to the data crawling task is installed on the device, this application can display the data crawling results and respond to user search requests. For example, if the business application is a smart search app, the electronic device can request the stored data crawling results from the server and display them to the user, or it can receive the user's search request, search for the relevant content from the data crawling results, and display it. In some implementations, if the user performs a refresh operation through the business application, the electronic device can request the latest data crawling results from the server. It is understandable that different business applications may support retrieving data crawling results from different storage media. Therefore, when configuring the storage media on the web client, different storage media can be configured to store the data crawling results according to the functions of different business applications, thus meeting the needs of different business applications.

[0095] Therefore, in the above Figure 1 Based on the system architecture, developers can dynamically configure the data crawling process on the web management page, which is more convenient and faster than the traditional way of writing code.

[0096] In other scenarios, the above system architecture can also be used to periodically retrieve some interface data. For example, by configuring the interface address and the content to be retrieved, the corresponding interface data can be automatically obtained.

[0097] Next, we'll introduce the configuration process for the data crawling process on the web client. Taking a personal computer (PC) as an example, for instance... Figure 2 As shown, the web client can log in and access the web management page, which could be the Smart Search Operation Cloud page. On the Smart Search Operation Cloud page, a crawler management control 21 is displayed in the left taskbar. Clicking the crawler management control 21 will display sub-function controls below it, including but not limited to the data crawling task management control 22, the URL management control 23, and the storage media management control 24. Clicking a control will display the corresponding configuration page on the right side of the Smart Search Operation Cloud page.

[0098] For example, when a developer clicks on the data crawling task management control 22, the web client displays the following: Figure 3The data crawling task management page shown provides functions including, but not limited to, querying and adding. The query function allows users to search for created data crawling tasks based on their name and / or code. The name can be the task name defined by the developers when creating the task (e.g., "Novel Ranking"), and the code can be the code defined by the developers when creating the task (e.g., "novel" corresponds to "novel"). Figure 3 As shown in the example, after entering "Novel Ranking" in the name input box 31 and clicking the query control 32, the currently created novel ranking data crawling tasks can be displayed on the current page. For the new feature, a new data crawling task can be created. Figure 4 As shown in the example, clicking the "Add Control" (41) will display the information required to add a new data crawling task on the current page. Developers fill in the name, code, description, data source, storage medium, crawling interval, etc., and then click the "Save Control" (42) to successfully create the data crawling task. The crawling interval here is a method for scheduled data crawling; that is, after the first data crawl is completed, a second data crawl can be performed after the set interval, and so on, for persistent scheduled data crawling.

[0099] For the process of filling in the data source when adding a new data crawling task, such as Figure 5 As shown, the location for filling in the data source can be displayed as a selection control 51. When the selection control 51 is clicked, a list of created URL data sources can be displayed on the current page, and developers can then select the URL data source to crawl from this list. For example Figure 5 The URL data sources to be crawled were selected as Hot List A and Hot List B.

[0100] For the process of filling in the storage medium when adding a new data crawling task, such as Figure 6 As shown, the location for filling in the storage medium can also be displayed in the form of a selection control 61. When the selection control 61 is clicked, a list of created storage media can be displayed on the current page, and the developer can then select at least one storage medium to save from the list. For example Figure 6The storage media list includes MySQL storage, Kafka storage, and Elasticsearch storage. The selected storage media are MySQL and Elasticsearch. MySQL is a relational database management system, well-suited for web applications. Kafka is a high-throughput distributed publish-subscribe messaging system primarily used for processing streaming data. It allows messages to be partitioned across multiple servers, providing redundancy and ensuring message persistence and high throughput. Elasticsearch is a document-based full-text search engine capable of real-time analysis and search of text data, offering efficient data retrieval and aggregation capabilities.

[0101] For ease of description, the following embodiments use these three storage media as examples. However, in real-world scenarios, more different types of storage media can be configured according to requirements, and this application embodiment does not impose any limitations on this.

[0102] After successfully creating a data crawling task, the task will be displayed on the data crawling task management page. Taking the created task as an example, which involves crawling a social trending list... Figure 7 As shown, this data crawling task record also corresponds to operation controls 71, which are used to trigger different operations such as task start, stop, and edit. If the developer clicks "Start Task," the data crawling task begins execution, and the web client sends a task request to the server for execution. If the developer clicks "Stop" after the task has started, the web client sends a stop request to the server to stop the data crawling task. If the developer clicks "Edit," the above settings will be re-displayed. Figure 4 The page allows developers to change the name, code, description, data source, storage medium, crawling time interval, and other information.

[0103] In some implementations, after successfully creating a data crawling task, the web client can generate a unique identifier for that task and send information such as the identifier, name, code, description, data source, storage medium, and crawling time interval to the server for storage. Optionally, the web client can also use the name of the data crawling task as its unique identifier. Optionally, the server can store the received information about the data crawling task in a first data table.

[0104] From the above Figure 5As described above, when adding a new data crawling task, a data source (i.e., a URL data source) needs to be selected. Therefore, the data source needs to be configured in advance for subsequent selection. The process of configuring the data source can be done through the above... Figure 2 The URL management control 23 is implemented on the corresponding page. For example, when a developer clicks the URL management control 23, the web client displays something like this: Figure 8 The URL management page shown provides functions including, but not limited to, querying and adding. For the query function, users can search for created URL data sources based on the name and / or URL address. The name can be the URL name defined by the developer when creating the URL data source (e.g., "Hot List A"), and the URL address can be the corresponding URL entered by the developer when creating the URL data source (e.g., the URL address corresponding to "Hot List A" is https: / / rebang.aaa). Figure 8 As shown in the example, after entering "Hot List A" in the name input box 81 and clicking the query control 82, the current page will display the URL data source information of the created Hot List A. For the new feature, a new URL data source can be created. Figure 9 As shown in the example, clicking the new control 91 will display the information required to add a new URL data source on the current page. After the developer fills in the name, code, description, URL address, type, etc., clicking the save control 92 will successfully create the URL data source.

[0105] In some implementations, after successfully creating a URL data source, the web client can generate a unique identifier for that URL data source and send information such as the identifier, name, encoding, description, URL address, and type of the URL data source to the server for storage. Optionally, the web client can also use the name of the URL data source as its unique identifier. Optionally, the server can store the received information about the URL data source in a second data table.

[0106] After successfully creating a URL data source, the created URL data source can be displayed on the URL management page. Taking the created URL data source as the data source for Hot List B as an example, ... Figure 10 As shown, this URL data source record also corresponds to operation control 93, which is used to trigger different operations such as editing, deleting, data extraction, and crawling data querying. If the developer clicks "edit," the web client will redisplay the above... Figure 9This page allows developers to modify information such as name, encoding, description, URL, and data type. If a developer clicks "Delete," the web client will delete the URL data source. Clicking "Data Extraction" allows developers to configure crawling rules for the URL data source. Since the data crawling task typically only needs to extract a portion of the content from the URL, configuring crawling rules is necessary to capture the required content.

[0107] The process of configuring crawling rules for the URL data source can be as follows: Figure 11 As shown, the R&D personnel in the above Figure 10 After clicking on the data extraction in operation control 93, the web client can jump to the display. Figure 11 The page shown displays the original source of the URL data source, crawler protocol restrictions, and the corresponding URL. It also displays a raw message retrieval control 94. Clicking this control allows the web client to send a request to the server, requesting the server to query the raw message of the URL data source's corresponding URL and display the data in the message information display box 95. Essentially, the message information displayed here is the complete content of the URL data source's corresponding URL. After configuring crawling rules, the desired content can be extracted from this message information.

[0108] After obtaining the raw message from the URL data source, developers can enter a rule expression in the crawling rule input box 96. In some implementations, the rule expression can be a regular expression (regex), which includes a combination of ordinary characters (e.g., letters from a to z) and special characters (called "metacharacters"). In other implementations, the rule expression can be XML Path Language (XPath), a language for determining the location of a part of an XML document. Developers can choose one of the rule expressions as needed. For example, Figure 11 Taking a regular expression as an example, after entering the rule expression, you can click the crawl test control 97 to test the crawling of raw messages based on the entered rule expression, thus verifying the accuracy of the entered rule expression. The information captured after the crawl test can be displayed in the crawled message information display box 98, for example... Figure 11As shown, the crawled results include field names, field values, and whether a field is required. Field names (e.g., title) can be automatically filled in by the web client based on rule expressions, or manually filled in by developers. Field values ​​are the titles of each message in the crawled raw message. The "required" option indicates whether the field value is required; if "yes" is selected, the field value is required. If the field value is empty during subsequent data crawling, the corresponding data crawling result can be discarded. If the crawling test results are normal, developers can click the save control 99. The web client will then send the crawling rules (including rule expressions and settings for required fields) and the crawling test results to the server for saving. It can be understood that the saved content corresponds to the identifier of the URL data source mentioned above; that is, the corresponding crawling rules can be found through the URL data source identifier for subsequent data crawling. Optionally, the server can store the crawling rules configured for the URL data source in a third data table, i.e., the third data table stores the configured rule expressions and settings for required fields. It is also understandable that the format and data of the original message of the URL data source may change. Therefore, the above rule expression can be adjusted periodically to crawl accurate message information.

[0109] From the above Figure 6 As described above, when adding a new data crawling task, it is also necessary to select a storage medium. Therefore, the storage medium must be configured in advance for subsequent selection. The process of configuring the storage medium can be done through the above... Figure 2 This is implemented through the page corresponding to the storage media management control 24, for example, such as... Figure 12 As shown, when a developer clicks on the storage media management control 24, two sub-controls can be displayed under the storage media management control 244: the storage source management control 241 and the storage media configuration control 242. The page corresponding to the storage source management control 241 is used to configure basic information about the storage source, and the configured storage source can be selected by the page corresponding to the storage media configuration control 242.

[0110] See also Figure 12 When the storage source management control 241 is clicked, a storage source management page appears on the web client. This page provides functions including, but not limited to, querying and adding. The query function allows users to search for created storage sources based on their name and / or code. The add function allows users to create a new storage source. Figure 12For example, clicking the "Add Control 1201" will display the information required to add a storage source on the current page, such as name, network type (or URL), hostname, username, password, and port. After filling in the information, clicking the "Save Control 1202" will successfully create the storage source, which can then be used as the storage server for the corresponding storage medium when configuring storage media later. This is understandable. Figure 12 The example shown is creating a MySQL storage source (e.g., named "Search mysql"). In addition, you can also create a Kafka storage source (e.g., named "Search kafka") and an Elasticsearch storage source (e.g., named "Search es").

[0111] When the storage media configuration control 242 is clicked, the web client displays the following: Figure 13 The storage media configuration page shown provides functions including, but not limited to, querying and adding. For the query function, users can search for created storage media based on the storage media's name and / or code. The name can be the storage media name defined by the developer when creating the storage media (e.g., a storage media named "Search Engine MySQL Storage"), and the code can be the code defined by the developer when creating the storage media (e.g., the code corresponding to "Search Engine MySQL Storage" is "hotMysql"). Figure 13 As shown in the example, after entering "Search Engine MySQL Storage" in the name input box 1301 and clicking the query control 1302, the current page will display the information of the created search engine MySQL storage media. For the "Add New" function, a new storage media can be created. Figure 14 As shown in the example, clicking the new control 1401 will display the information that needs to be filled in when adding a new storage medium on the current page. After the R&D personnel fill in the name, code, description and other information, they can click the save control 1402 to successfully create the storage medium.

[0112] In some implementations, after successfully creating the storage medium, the web client can generate a unique identifier for that storage medium and send its identifier, name, encoding, description, and other information to the server for storage. Optionally, the web client can also use the name of the storage medium as its unique identifier. Optionally, the server can store the received information about the storage medium in a fourth data table.

[0113] After successfully creating the storage media, it will be displayed on the storage media configuration page. For example, if the created storage media include MySQL storage for the search engine, Kafka storage for the search engine, and Elasticsearch storage for the search engine, then... Figure 15As shown, each record on the storage medium also corresponds to an operation control 1501 and a "clickable configuration" control 1502. The operation control 1501 is used to trigger different operations such as editing and deleting. If the developer clicks "edit," the web client will redisplay the above-mentioned controls. Figure 14 The page allows developers to change the name, code, description, and other information. If a developer clicks "Delete," the web client will delete the record for that storage medium. The connection configuration process for the storage medium can be implemented using configuration control 1502. When a configuration control 1502 is clicked, the web client can display... Figure 16 The storage connection configuration page shown.

[0114] because Figure 15 It contains three different types of storage media, and the connection configuration methods corresponding to the different types of storage media are different. Therefore, in Figure 16 The storage connection configuration page allows for configuration of different types of storage media. For example, the storage connection configuration page may display a storage source selection control 1601. Developers can click this control to select a storage source from the list to configure the current storage medium; the selected storage source here is the one mentioned above. Figure 12 The storage source configured in [the configuration]. Where, if... Figure 16 By clicking Figure 15 If you are redirected from the Chinese search engine's MySQL storage configuration control, you can then select to search for MySQL in the storage source list; if Figure 16 By clicking Figure 15 If you are redirected from the Kafka storage configuration control in the Chinese search engine, you can then select Kafka from the storage source list; if Figure 16 By clicking Figure 15 If you are redirected from the configuration controls for the Elasticsearch (ES) storage source, you can then select ES from the storage source list. For the case where you select MySQL as the storage source, as shown below... Figure 17As shown, after the developers select "MySQL" as the search term, the current page displays the storage source information for the searched MySQL database, such as hostname, username, password, and port. A connection test control 1701 is also displayed. Clicking this control tests the network connectivity with the storage server. If the network is connected, a "Connection Successful" message is displayed. After a successful connection test, the developers can enter the configured MySQL statement in the MySQL configuration input box 1702, such as a statement to create a data table. This created table will be the storage medium for storing the crawled data results. Then, by clicking the DDL execution control 1703, the developers can request the server to execute the command to create the data table. If the server successfully creates the data table, it returns a success message to the web client, which displays a "Execution Successful" message on the current page.

[0115] For cases where the storage source is selected to search Kafka, such as Figure 18 As shown, after the developer selects Kafka for search, the current page displays the storage source information for the searched Kafka, such as the hostname. A connection test control 1801 is also displayed. Clicking this control tests the network connectivity with the storage server. If the network is connected, a "Connection Successful" message is displayed. After a successful connection test, the developer can enter the configured topic name in the Kafka configuration input box 1802. This created topic is a logical channel for messages. Producers store messages (or data) in the topic, and consumers can subscribe to messages (or data) from the topic. This topic serves as the storage medium for storing the data crawling results. Then, clicking the create topic control 1803 requests the server to execute a topic creation command. If the server successfully creates the topic, it returns a success message to the web client, which displays a "Creation Successful" message on the current page.

[0116] For cases where Elasticsearch is selected as the storage source, such as... Figure 19As shown, after the developer selects Elasticsearch (ES) as the search target, the current page displays the storage source information for the ES search, such as hostname, username, and password. A connection test control (1901) is also displayed. Clicking this control tests the network connectivity with the storage server. If the network is connected, a "Connection Successful" message is displayed. After a successful connection test, the developer can enter the configured DSL (Domain Specific Language) statements in the ES configuration input box (1902) to create an ES database index, which will be used to save crawled data results later. Then, clicking the create control (1903) requests the server to execute the index creation command. If the server successfully creates the index, a success message is returned to the web client, which displays a "Created Successfully" message on the current page.

[0117] In some implementations, after configuring the connection settings for each storage medium, the web client can send the connection configuration information to the server for storage. Optionally, the server can store the received connection configuration information for the storage media in a fifth data table.

[0118] It is understandable that, regardless of which of the above storage sources is chosen as the storage medium, the data crawling results must be parsed into the corresponding data format before being stored accordingly.

[0119] The above configuration process completes the setup for the URL data source, storage medium, and data crawling task. Then, if the developer clicks the above... Figure 7 Once the task in the operation control 71 begins, the web client will send a task request to the server so that the server can execute the corresponding data crawling task.

[0120] Next, we will introduce the detailed process of the server executing data crawling tasks. For example... Figure 20 As shown, the process may include:

[0121] S11, the server receives a data crawling task request sent by the web client.

[0122] The task request can carry an identifier corresponding to the data crawling task, such as the task ID or the name of the data crawling task.

[0123] S12, the server starts the data crawling service and starts a persistent thread.

[0124] Since a crawling time interval was set when creating the data crawling task, meaning that data crawling needs to be performed continuously, the server needs to start a persistent thread to continuously retrieve the data source to be crawled from the crawling queue.

[0125] In some implementations, the server can use the @construct and @postconstruct annotations to start the persistent thread, and use a while(true) conditional logic to make the persistent thread loop.

[0126] S13, the server locks the data crawling service. If the locking is successful, then S15 is executed.

[0127] In a real-world scenario, there may be multiple servers deployed, and different servers can be used to execute different service processes. Therefore, if you want the data crawling service on the current server to execute the current data crawling task and prevent other servers or other services from repeatedly executing the current data crawling task, you can lock the current data crawling service.

[0128] In some implementations, the server can add both local and distributed locks to the data crawling service. Local locks prevent other services from concurrently reading and writing data corresponding to the crawling task, thus avoiding data inconsistency. Distributed locks prevent multiple services from repeatedly executing the current data crawling task. Optionally, the server can use the `setnx` command to acquire the distributed lock and set its expiration time to N, for example, N = 1 minute. Setting the expiration time here prevents deadlocks; if the current server encounters an error but holds the lock, other servers will not execute the corresponding data crawling service, causing the data crawling task to fail. By setting the expiration time, the server can unlock the lock after N minutes, allowing the data crawling task to continue.

[0129] S14, the server obtains the information of the data source to be crawled based on the identifier corresponding to the data crawling task in the task request, and stores it in the crawling queue.

[0130] It should be noted that the process of obtaining the data source information to be crawled in S14 can be carried out simultaneously with the processes of starting the data crawling service in S12-S13, that is, the server can perform multiple preparatory work for the data crawling task at the same time.

[0131] In S14, because the task request received by the server carries an identifier corresponding to the data crawling task, and the web client, after successfully creating the data crawling task, has sent information such as the identifier, name, code, description, crawling data source, and crawling time interval of the data crawling task to the server for storage, the server can find the corresponding crawling data source (i.e., the data source to be crawled) based on the identifier of the data crawling task. Furthermore, because the web client, after successfully creating the URL data source, has also sent information such as the identifier, name, code, description, URL address, and type of the URL data source to the server for storage, the server can find the identifier and URL address of the crawling data source corresponding to the data crawling task. For example, the server can find the corresponding crawling data source (i.e., the URL data source) from the first data table based on the identifier of the data crawling task, and then find the corresponding URL address from the second data table based on the identifier of the URL data source, thereby linking the data crawling task, the URL data source, and the URL address. Therefore, the server can obtain information about the data source to be crawled based on the identifier corresponding to the data crawling task in the task request. This information includes, but is not limited to, the data source identifier, URL, and crawl trigger time. The server then stores this information in a crawling queue. For example, the data crawling task is related to the above... Figure 6 The task of crawling the social hot list has two data sources to be crawled: Hot List A and Hot List B. The server can store the information corresponding to the two data sources, Hot List A and Hot List B, into the crawl queue.

[0132] Here, the crawling trigger time can be determined by the system time plus the crawling interval. For example, if the system time when the server receives the task request is 10:00:00 and the crawling interval is 2 minutes, then the corresponding data crawling task's crawling trigger time should be 10:02:00, and the server can then crawl the data according to this crawling trigger time.

[0133] In some implementations, the queue to be crawled can be a Redis sorted set (zset) queue. A zset queue is an ordered queue in the Redis database. When the server stores information about the data source to be crawled into the zset queue, it can use the data source identifier as the value and the crawl trigger time as the score, and store the data in order of crawl trigger time. For example, the information about the data source to be crawled can be stored into the zset queue in order of crawl trigger time from earliest to latest.

[0134] It's understandable that since developers can trigger multiple different data crawling tasks at different times on the web client, the web client needs to send multiple task requests to the server to request the initiation of different data crawling tasks. Therefore, the crawling queue will store information about the data sources corresponding to different data crawling tasks, and a single data crawling task can have multiple data sources. For example, if the web client starts three data crawling tasks: the first task corresponds to one data source (A) with a crawling trigger time of 10:00:00; the second task corresponds to two data sources (B and C) with a crawling trigger time of 10:05:00; and the third task corresponds to two data sources (D and E) with a crawling trigger time of 10:10:00, then the information of data sources A, B, C, D, and E will all be stored in the crawling queue.

[0135] S15, the server retrieves a preset number of data sources to be crawled from the crawl queue.

[0136] Because the persistent threads started by the server continuously crawl data, the queue of data sources to be crawled will accumulate more and more. If all data sources to be crawled were crawled simultaneously, it would undoubtedly increase the server's processing load and cause excessive overload. Therefore, the server can retrieve a preset number of data sources to be crawled from the queue each time. For example, the preset number could be 20.

[0137] In some implementations, when the queue to be crawled is a zset queue, the server can retrieve the first 20 data sources to be crawled in the zset queue each time, that is, the first 20 data sources to be crawled after being sorted from earliest to latest according to the crawling trigger time.

[0138] S16, the server determines whether the current system time has reached the crawling trigger time of the first data source to be crawled. If it has, then execute S17; otherwise, execute S21.

[0139] The first data source to be crawled is any one of the preset number of data sources to be crawled mentioned above.

[0140] As described above, each data source to be crawled has a corresponding crawling trigger time. The server will only start crawling the corresponding data source when the corresponding crawling trigger time is reached. Therefore, the server will determine whether the current system time has reached the crawling trigger time of the first data source to be crawled.

[0141] In some implementations, the server can subtract the crawl trigger time of the first data source to be crawled from the current system time to obtain the time difference. If the time difference is greater than or equal to 0, it means that the current system time has reached the crawl trigger time of the first data source to be crawled, and the server needs to start the data crawling process. If the time difference is less than 0, it means that the current system time has not yet reached the crawl trigger time of the first data source to be crawled, and the server can continue to wait.

[0142] It is understandable that since the server obtains a preset number of data sources to be crawled, it is necessary to determine whether the crawling trigger time of each data source has been reached. For multiple data sources that have reached the crawling trigger time, step S17 can be executed, and for multiple data sources that have not reached the crawling trigger time, step S21 can be executed.

[0143] S17, the server calculates the first waiting time for the first data source to be crawled.

[0144] Here, the first data source to be crawled at this time is the data source to be crawled that has reached the crawling trigger time. The first waiting time waittime can be obtained by subtracting the current system time from the crawling trigger time. The calculated first waiting time is used for the subsequent control of the local lock.

[0145] It is understandable that since the current system time has reached the crawling trigger time of the first data source to be crawled, the first waiting time waittime will be less than or equal to 0. Subsequently, the absolute value of waittime can be used to control the local lock time.

[0146] S18, the server stores the URL corresponding to the first data source to be crawled into the data crawling queue for the crawling engine to crawl.

[0147] That is, for each first data source to be crawled that reaches the crawling trigger time, the server can first store the URLs corresponding to these data sources into the data crawling queue, so that the crawler engine can read the URLs from the data crawling queue in sequence to crawl the data.

[0148] For details on the data crawling process performed by the web crawler engine, please refer to the following: Figure 21 The description will not be repeated here.

[0149] S19, the server removes the information of the first data source to be crawled from the crawl queue, and determines the next crawl trigger time for the first data source to be crawled.

[0150] S20, the server re-stores the update information of the first data source to be crawled into the crawl queue, and then executes S22.

[0151] Since the server has already stored the URL corresponding to the first data source to be crawled in the data crawling queue, and the crawler engine will subsequently execute this data crawling task, the server can delete the information of the first data source to be crawled at this time. After determining the next crawling time, the updated information of the first data source to be crawled will be stored back in the crawling queue for the next crawling process. This enables a continuous, automatic, and scheduled data crawling process. It can be understood that the updated information of the first data source to be crawled includes, but is not limited to, the data source identifier, URL, and the next crawling trigger time.

[0152] In some implementations, the server determines the next crawl trigger time for the first data source to be crawled by: obtaining the crawling time interval of the data crawling task corresponding to the first data source to be crawled, and then adding the corresponding crawling time interval to the current system time, which gives the next crawl trigger time. Therefore, the above continuous data crawling process can also be understood as a timed data crawling process, with the time interval being the duration of the crawling interval.

[0153] It is also understandable that for each first data source to be crawled, the server will execute the steps S18-S20 above.

[0154] In some implementations, if the web client does not configure a crawling time interval for the aforementioned data crawling task, the server can also set a default crawling time interval to periodically and continuously perform the data crawling process.

[0155] S21, the server calculates the second waiting time for the first data source to be crawled.

[0156] Here, the first data source to be crawled is the data source to be crawled that has not yet reached the crawling trigger time. The second waiting time can be obtained by subtracting the current system time from the crawling trigger time. The calculated second waiting time is used for the subsequent control of the local lock.

[0157] S22, the server deletes the distributed lock.

[0158] The purpose of deleting the distributed lock here is to prevent deadlocks caused by the current service continuously holding the distributed lock. For example, if a data crawling task is waiting to be executed before its trigger time, and the corresponding service holds the distributed lock, a new data crawling task might fail to execute if it enters the system. Therefore, the server can delete the distributed lock and re-lock it during the next data crawling process.

[0159] S23, the server determines whether the first waiting time or the second waiting time is greater than the first preset duration. If it is greater, then execute S24; otherwise, execute S25.

[0160] S24, the server executes the local lock for the first preset duration, then releases the lock and returns to S13.

[0161] S25, the server executes the first or second waiting time for the local lock, and then releases the lock and returns to S13.

[0162] In other words, while waiting for the next data crawling session, the server will execute a local lock based on either a first or second waiting time. To prevent the next execution time from being later than the crawling trigger time, it will determine whether the waiting time is greater than a first preset duration (e.g., the first preset duration is 1 minute). If it is greater than the first preset duration, the server will lock the local lock for the first preset duration to avoid wasting resources due to prolonged locking. If it is not greater than the first preset duration, the server will lock the local lock for either the first or second waiting time to execute the next round of crawling tasks as soon as possible. After executing and releasing the local lock, the server will return to step S13 above to automatically start the next round of crawling tasks.

[0163] For example, suppose the crawling trigger time for data crawling task A is 10:35:00, and the current system time is 10:30:30, meaning the current system time has not yet reached the crawling trigger time. The calculated second waiting time is 4 minutes and 30 seconds. If the second waiting time is greater than 1 minute (assuming the first preset duration is 1 minute), the server locally locks for 1 minute. Then, the lock is released, and execution returns to S13, restarting the next data crawling process. Since data crawling task A has not yet been executed, the next data crawling process will again check data crawling task A. After the aforementioned 1-minute local locking, the current system time should reach 10:31:30. Since the crawling trigger time has still not been reached, the calculated second waiting time is 3 minutes and 30 seconds, requiring another 1-minute local locking, and so on. This continues until the current system time reaches 10:34:30, at which point the calculated second waiting time is 30 seconds. Since the second waiting time is less than 1 minute, the server locally locks for 30 seconds, then releases the lock and returns to execution to S13. Since the current system time is 10:35:00, the crawling trigger time has been reached. The server can then store the URL corresponding to the data crawling task A into the data crawling queue for the crawling engine to crawl.

[0164] In some implementations, the steps described above performed by the server can be executed by a scheduled task engine running within the server.

[0165] In step S18 above, after the server stores the URL corresponding to the second data source to be crawled into the data crawling queue, the crawler engine (running within the server) can start crawling once it recognizes that data has been stored in the data crawling queue. For example... Figure 21 As shown, taking the server as the execution entity as an example, this data crawling process may include:

[0166] S31, the server starts a persistent thread.

[0167] It should be noted that the persistent thread started here is not the same persistent thread in S12 above. The persistent thread in S12 is used to continuously retrieve the data source to be crawled from the crawl queue, so as to store the URL of the data source to be crawled that has reached the crawl trigger time into the data crawl queue. The persistent thread in S31 is used to continuously crawl data according to the URL.

[0168] S32, the server initializes the web page downloader and download thread pool.

[0169] A webpage downloader is a data downloader used to download webpage data corresponding to a URL. The server can initialize the webpage downloader according to the specific data crawling business or use a default webpage downloader. A download thread pool is used for concurrent downloading of webpage data, meaning it can concurrently download webpage data corresponding to different URLs. The server can initialize the download thread pool according to the specific data crawling business or create a default download thread pool.

[0170] S33, the server determines whether the download thread pool is in a running state. If it is, it executes S34; otherwise, the process ends.

[0171] If the download thread pool is in a normal (running) state, the server can execute the subsequent data crawling process. If the download thread pool is not running (it may be due to an error or stop running), the current process will end.

[0172] S34, the server retrieves the URL of the first data source to be crawled from the data crawling queue.

[0173] S35, determine if the obtained URL is empty. If it is empty, execute S36; otherwise, execute S37.

[0174] Since the server has stored the URL corresponding to the first data source to be crawled in the data crawling queue, it can now retrieve the URL from this queue. Considering potential anomalies such as read / write errors, the server might obtain an empty URL. Therefore, the server can check if the obtained URL is empty. If it is not empty, the data crawling process can proceed normally; if it is empty, the current round of data crawling cannot be executed, and the next round can begin. For example, the URL obtained by the server might be the URL of the aforementioned hot list A.

[0175] S36, the server executes the local lock for the second preset duration, then releases the lock and returns to S33.

[0176] If the obtained URL is empty, the server can execute a local lock for a second preset duration to avoid frequent iterations of the next loop, thus avoiding resource waste. In some implementations, the second preset duration can be 1 second.

[0177] It is understandable that after the server releases the lock and returns S33, when retrieving the URL from the data crawling queue, the URL obtained will be the next new URL.

[0178] S37, the server submits download tasks through the download thread pool to download the web page data corresponding to the URL.

[0179] If the obtained URL is not empty, the server can trigger the download of the webpage data corresponding to the URL. Here, the download task can be submitted through the download thread pool mentioned above to download the webpage data corresponding to the URL.

[0180] In some implementations, the maximum and minimum number of threads in the download thread pool can be 2C+1, where C is the number of CPU cores on the server. Setting the same maximum and minimum number of threads maximizes CPU resource utilization. The maximum length of the download task queue in the download thread pool can be M (e.g., M = 1000) to ensure that download tasks are submitted successfully as much as possible. If a download task submission fails, the server can record the corresponding download task, for example, in a data table, and then reprocess the download task in that data table later.

[0181] In some implementations, the process of downloading the webpage data corresponding to the URL can be performed by a webpage downloader, and the specific process may include:

[0182] A: A webpage downloader that retrieves download links.

[0183] The download linker is a long-lived connection associated with a webpage domain. You can download the data you want to obtain through the download linker. For example, the download linker can be implemented using an HTTP connection pool.

[0184] B: Set the HTTP request message.

[0185] The HTTP request message can carry information such as request headers, request body, socket timeout, and connection timeout. The request header is used to describe the metadata of the HTTP request message, including information such as the request method and request protocol. The request body is used to pass the parameters or content required for the HTTP request to the web server.

[0186] C: Request web page data from the web server based on the HTTP request message.

[0187] D: Obtain the webpage data stream and convert the data stream into a text string.

[0188] After setting up the HTTP request message, the web page downloader can request the corresponding web page data from the web server. Generally, the initially obtained web page data is in web page data stream format, and the data stream can also be converted into a text string.

[0189] S38, the server obtains the crawling rules corresponding to the first data source to be crawled.

[0190] Due to the above Figure 11 The system configures crawling rules for URL data sources. The first data source to be crawled is one of the configured URL data sources, allowing the server to retrieve the crawling rules corresponding to that first data source. In some implementations, after the web client sends crawling rules and other information about the URL data source to the server, the server stores this information in a third-party table. The server can then retrieve the crawling rules corresponding to the first data source from this third-party table.

[0191] S39: Determine if the crawling rule is empty. If it is empty, end the process. If it is not empty, execute S40.

[0192] Considering scenarios such as read / write errors or incomplete configuration of crawling rules for URL data sources, the process can end if the crawling rules obtained by the server are empty, and can proceed with the subsequent rule matching process if the crawling rules are not empty.

[0193] S40, the server matches webpage data according to the crawling rules.

[0194] The server can use the rule expressions in the crawling rules (such as those mentioned above) to crawl the web. Figure 11 The regular expression entered in the input field is used to match the obtained web page data.

[0195] S41, determine whether the crawling rule matches. If it does not match, end the process. If it matches, execute S42.

[0196] That is, it determines whether the crawling rules used in S40 match the data. If they match, the crawled data is then processed.

[0197] S42, the server continuously retrieves matching data and retrieves data corresponding to required fields according to the crawling rules.

[0198] Because the amount of data corresponding to a single URL can be large, the server can repeatedly retrieve matching data to reduce the probability of missing data. In some implementations, after retrieving the matching data, the server can also concatenate the matching data according to the order of the expressions in the aforementioned rule expression, for example, concatenating the matching data in the order of title and source. Then, the server can also retrieve the aforementioned... Figure 11 The settings for whether fields are required in the configured crawling rules are used to obtain the data corresponding to the required fields in the matched data.

[0199] S43, determine if the matching data is empty. If it is empty, end the process. If it is not empty, execute S44.

[0200] S44: Determine if the data corresponding to the required field is empty. If it is empty, end the process. If it is not empty, execute S45.

[0201] In other words, the server will check whether the obtained matching data and the data corresponding to the required fields are empty. If any data is empty, the record can be deleted, and the current process ends. If neither the matching data nor the data corresponding to the required fields is empty, the subsequent steps will be executed.

[0202] S45, after the server processes the matching data, it saves it to the storage medium.

[0203] For example, in scenarios where the matching data is a text string, the server can convert it to JSON format for storage. In some implementations, the server can also fill in and concatenate the matching data into a complete sentence before saving it. For instance, in the case of crawling the aforementioned social trending topics list, information such as title and source can be filled in and concatenated into a complete sentence. Because the web client is configured with the storage medium corresponding to the data crawling task, the server can store the matching data (i.e., the crawled web page data) according to the configured storage medium. The specific process can be found below. Figure 22 The example shown.

[0204] In addition, in some implementations, after the server has finished crawling the web page data corresponding to the current URL, it can also determine whether there is child node information under the URL, that is, whether there is a child page. If so, it can also obtain the URL of the child page and continue to crawl the data of the child page, and also save the crawled data in the storage medium.

[0205] In some implementations, the process of analyzing the downloaded webpage data in S38-S45 described above can be performed by a webpage parser on the server.

[0206] It is understandable that after completing this round of data crawling, the server can return to S33 to continue the persistent data crawling process by retrieving URLs from the data crawling queue.

[0207] The above data crawling method allows for dynamic configuration of data crawling tasks on the web client, reducing the human cost of writing code. At the same time, the server can continuously perform the data crawling process according to the crawling time interval, improving the efficiency and flexibility of data crawling.

[0208] Regarding the process of storing the crawled webpage data in S45 above, as follows: Figure 22 As shown, the process may specifically include the following steps:

[0209] S51, the server obtains the information of the first storage medium corresponding to the data crawling task based on the identifier corresponding to the data crawling task.

[0210] The identifier corresponding to the data crawling task is the content carried in the task request. The server can obtain the information of the first storage medium from the configured information based on the identifier of the data crawling task.

[0211] Since the server already stores information such as the identifier, name, code, description, data source, storage medium, and crawling time interval corresponding to the data crawling task, the server can find the corresponding first storage medium based on the identifier of the data crawling task. Furthermore, since the web client has also sent the identifier, name, code, and description of the storage medium to the server for storage after successfully creating it, the server can find the information corresponding to the first storage medium. For example, the server can find the corresponding first storage medium from the first data table based on the identifier of the data crawling task, and then find the information corresponding to that first storage medium from the fourth data table based on the found first storage medium.

[0212] It is understandable that there can be one or more first storage media corresponding to the configured data crawling task. When there are multiple first storage media, the server can obtain the information of each first storage media separately.

[0213] S52, determine whether the information of the first storage medium is empty. If it is empty, end the process. If it is not empty, execute S53.

[0214] Considering scenarios such as read / write errors or incomplete configuration of storage media information, if the information of the first storage media obtained by the server is empty, the process can be terminated; if the information of the first storage media is not empty, subsequent storage procedures can be executed.

[0215] S53, obtain the connection configuration information corresponding to the first storage medium.

[0216] Since the web client sends the corresponding connection configuration information to the server for storage after completing the connection configuration of the storage medium, the server can also obtain the connection configuration information corresponding to the first storage medium. For example, the server can obtain the connection configuration information corresponding to the first storage medium from the fifth data table. For the MySQL storage medium of the search engine, the corresponding connection configuration information may include hostname, username, password, and port information. For the Kafka storage medium of the search engine, the corresponding connection configuration information may include hostname and topic name information. For the Elasticsearch storage medium of the search engine, the corresponding connection configuration information may include hostname, username, and password information.

[0217] S54, the server determines the type of the first storage medium.

[0218] For example, taking the aforementioned search engine MySQL storage, search engine Kafka storage, and search engine Elasticsearch storage as examples, the type of the first storage medium can be one or more of the following types: MySQL, Kafka, and Elasticsearch.

[0219] S55, if the type of the first storage medium includes MySQL, determine whether the corresponding connection configuration information is empty. If it is empty, end the process; otherwise, execute S56.

[0220] Considering scenarios where some data crawling tasks configure storage media but fail to connect to it, the storage media is not connected to the storage server, making it impossible to store data. Therefore, the server can check if the connection configuration information for the first storage media is empty (or missing). If empty, the process ends; otherwise, subsequent steps are executed. For example, the server checks if the MySQL hostname, username, password, and port are empty.

[0221] S56, the server creates a MySQL connection client.

[0222] Therefore, if the connection configuration information of the MySQL storage medium of the search engine is complete, the server can store the crawled web page data into the storage medium. However, before storage, it is necessary to establish a connection with the storage medium, that is, to create a MySQL connection client. Through the connection client, data can be stored into the corresponding storage medium.

[0223] S57: Determine if the MySQL connection client was created successfully. If not, end the process; if successful, execute S58.

[0224] S58 stores the information of the MySQL connection client for direct use when storing data later.

[0225] In other words, once a MySQL connection client is successfully created, the server can save the MySQL connection client information to facilitate continuous data storage. In one implementation, the server can store the MySQL connection client information in a map container. When data needs to be stored on the corresponding storage medium later, lazy loading can be used to retrieve the MySQL connection client information directly from the map container and store the data on the corresponding search engine's MySQL storage medium.

[0226] S59, the server saves the crawled web page data to the search engine's MySQL storage.

[0227] Because the connection configuration page of the aforementioned web client contains MySQL statements for creating data tables, the server can store web page data in the created data tables.

[0228] S60, if the type of the first storage medium includes Kafka, determine whether the corresponding connection configuration information is empty. If it is empty, end the process; otherwise, execute S61.

[0229] Similar to the MySQL-type process described above, the server can also determine whether the connection configuration information for the first storage medium is empty (or missing). If it is empty, the process ends; otherwise, subsequent steps are executed. For example, the server can check whether the Kafka hostname and topic name are empty.

[0230] S61, the server creates a Kafka connection client.

[0231] Therefore, if the connection configuration information of the Kafka storage medium of the search engine is complete, the server can store the crawled web page data into the storage medium. However, before storage, it is necessary to establish a connection with the storage medium, that is, to create a Kafka connection client. Through the connection client, data can be stored into the corresponding storage medium.

[0232] S62 checks whether the Kafka connection client has been successfully created. If it has not been created, the process ends. If it has been created, S63 is executed.

[0233] S63 stores the information about the Kafka connection client so that it can be used directly when storing data later.

[0234] In other words, once a Kafka connection client is successfully created, the server can save the Kafka connection client information to facilitate continuous data storage. In one implementation, the server can store the Kafka connection client information in a map container. When data needs to be stored on the corresponding storage medium later, lazy loading can be used to retrieve the Kafka connection client information directly from the map container and store the data on the corresponding Kafka storage medium.

[0235] S64, the server saves the crawled web page data to the search engine's Kafka storage.

[0236] Since the topic name is configured on the connection configuration page of the aforementioned web client, the server can store the web page data in the corresponding topic.

[0237] S65, if the type of the first storage medium includes the es type, determine whether the corresponding connection configuration information is empty. If it is empty, end the process; otherwise, execute S66.

[0238] Similar to the processes described above for MySQL and Kafka types, the server can also check if the connection configuration information for the first storage medium is empty (or missing). If it is empty, the process ends; otherwise, subsequent steps are executed. For example, the server can check if the Elasticsearch hostname, username, password, and other information are empty.

[0239] S66, the server creates an Elasticsearch connection to the client.

[0240] Therefore, if the connection configuration information of the Elasticsearch (ES) storage medium is complete, the server can store the crawled web page data into the storage medium. However, before storage, it is necessary to establish a connection with the storage medium, that is, to create an ES connection client. Through this connection client, data can be stored into the corresponding storage medium.

[0241] S67, determine whether the Elasticsearch client connection was created successfully. If not, end the process; if successful, execute S68.

[0242] S68 stores the information of the Elasticsearch client connection for direct use when storing data later.

[0243] In other words, once an Elasticsearch (ES) connection client is successfully created, the server can save the client's information to facilitate continuous data storage. In one implementation, the server can store the client's information in a map container. When data needs to be stored on the corresponding storage medium later, lazy loading can be used to retrieve the client's information directly from the map container and store the data on the corresponding Elasticsearch storage medium.

[0244] S69, the server saves the crawled web page data to the search engine's Elasticsearch storage.

[0245] Because the connection configuration page of the aforementioned web client contains DSL statements and creates Elasticsearch database indexes, the server can store web page data in the corresponding Elasticsearch database.

[0246] It's understandable that since there may be multiple primary storage media for the configured data crawling tasks, the server will store the crawled webpage data into each primary storage media separately. Subsequently, if a business application needs to obtain webpage data, it can search for and retrieve the data from the storage media subscribed to or bound to that business application, thus meeting the needs of different business applications. Furthermore, if a new storage media for storing webpage data needs to be added, it can be done directly in the aforementioned... Figure 7 On the page shown, click "Edit" in operation control 71 to change the storage medium configuration.

[0247] It is also understandable that if one of the multiple configured primary storage media fails to save web page data, the server can record the failure log, and the failure record does not affect the storage information of other storage media.

[0248] It's understandable, based on the above. Figure 1 The system architecture shown allows the server to retrieve data and store the results in a storage medium. If the user's electronic device has a business application installed that corresponds to the data retrieval task, the application can retrieve the data retrieval results from the storage medium, display the results, and respond to the user's search requests.

[0249] The above data crawling method reduces the human cost of writing code by dynamically configuring the storage medium for saving the data crawling results on the web client. At the same time, the dynamic configuration method has high flexibility, which can meet the needs of different business applications and improve the efficiency of data crawling.

[0250] In some scenarios, the process of dynamically configuring the data crawling task described above can be executed not only on the web client but also on the server, meaning developers can directly access the web management page on the server to configure it. In other scenarios, when the web client has sufficient computing power, the data crawling process can also be executed on the web client.

[0251] The aforementioned web client can be a device such as a laptop or PC, for example, Figure 23 This is a schematic diagram of the structure of a web client provided in an embodiment of this application. The web client may include a processor 210, a memory 220, and a communication module 230, etc.

[0252] The processor 210 may include one or more processing units, and the memory 220 is used to store program code and data. In this embodiment, the processor 210 can execute computer execution instructions stored in the memory 220.

[0253] The communication module 230 can be used for communication between various internal modules of the web client, or for communication between the web client and other devices. For example, if the web client communicates with other devices via a wired connection, the communication module 230 may include an interface, such as a Universal Serial Bus (USB) interface. The USB interface can be an interface conforming to the USB standard specification, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface can be used to connect a charger to charge the web client, to transfer data between the web client and other devices, and to connect headphones for audio playback, etc.

[0254] Alternatively, the communication module 230 may include audio devices, radio frequency circuits, Bluetooth chips, wireless fidelity (Wi-Fi) chips, near-field communication (NFC) modules, etc., which can enable interaction between the web client and other devices in a variety of different ways.

[0255] Additionally, the web client may include a display screen 240, which can display images or videos from the human-computer interaction interface. For example, it can display the aforementioned web management page, allowing developers to configure data crawling tasks.

[0256] Optionally, the web client may also include peripheral devices 250, such as a mouse, keyboard, speaker, microphone, etc.

[0257] It should be understood that, in addition to Figure 23 Beyond the various components or modules listed, the embodiments of this application do not specifically limit the structure of the web client. In other embodiments of this application, the web client may also include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The components illustrated can be implemented in hardware, software, or a combination of software and hardware.

[0258] The aforementioned server can be a single server or a server cluster, for example, Figure 24This is a schematic diagram of a server structure provided in an embodiment of this application. The server includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The server's database can be used to store configuration data for the aforementioned data crawling tasks, configuration data for URL data sources, etc. The server's network interface can be used to communicate with external devices via a network connection.

[0259] The aforementioned electronic devices can be user-used mobile phones, tablets, etc., for example, Figure 25 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Taking a mobile phone as an example, the electronic device may include a processor 110, an external memory interface 120, an internal memory 121, a USB interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identity module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0260] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0261] The controller can serve as the nerve center and command center of an electronic device. Based on the instruction opcode and timing signals, the controller generates operation control signals to control the fetching and execution of instructions.

[0262] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0263] The mobile communication module 150 can provide solutions for wireless communication applications including 2G / 3G / 4G / 5G in electronic devices. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0264] The wireless communication module 160 can provide solutions for wireless communication applications in electronic devices, including wireless local area networks (WLANs) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0265] Electronic devices implement display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connecting the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0266] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, the electronic device may include one or N displays 194, where N is a positive integer greater than 1.

[0267] Internal memory 121 can be used to store executable program code, including instructions. Processor 110 executes various functional applications and data processing of the electronic device by running the instructions stored in internal memory 121. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback, image playback, etc.), etc. The data storage area may store data created during the use of the electronic device (such as audio data, phonebook, etc.). Furthermore, internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0268] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device. In other embodiments of this application, the electronic device may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0269] The foregoing has detailed examples of the data crawling methods provided in the embodiments of this application. It is understood that web clients, servers, and electronic devices, in order to achieve the above functions, include corresponding hardware and / or software modules for performing each function. Those skilled in the art should readily recognize that, based on the units and algorithm steps of the examples described in conjunction with the embodiments disclosed herein, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed in hardware or by computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application in conjunction with the embodiments, but such implementation should not be considered beyond the scope of this application.

[0270] This application embodiment can divide the Web client, server, and electronic device into functional modules based on the above method examples. For example, each function can be divided into its own functional module, such as a detection unit, a processing unit, and a display unit. Alternatively, two or more functions can be integrated into one module. The integrated module can be implemented in hardware or as a software functional module. It should be noted that the module division in this application embodiment is illustrative and only represents one logical functional division; other division methods may be used in actual implementation.

[0271] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0272] When using integrated units, the electronic device may further include a processing module, a storage module, and a communication module. The processing module is used to control and manage the operation of the electronic device. The storage module supports the execution of stored program code and data. The communication module supports communication between the electronic device and other devices.

[0273] The processing module can be a processor or a controller. It can implement or execute various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. The processor can also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a digital signal processor and a microprocessor, etc. The storage module can be a memory. The communication module can specifically be a radio frequency circuit, a Bluetooth chip, a Wi-Fi chip, or other devices that interact with other electronic devices.

[0274] This application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor performs the data crawling method of any of the above embodiments. The storage medium may include various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0275] This application also provides a computer program product that, when run on a computer, causes the computer to perform the aforementioned steps to implement the data crawling method described in the above embodiments.

[0276] In addition, embodiments of this application also provide an apparatus, which may specifically be a chip, component, or module. The apparatus may include a connected processor and a memory; wherein the memory is used to store computer execution instructions, and when the apparatus is running, the processor may execute the computer execution instructions stored in the memory to cause the chip to execute the data crawling methods in the above-described method embodiments.

[0277] In this embodiment, the Web client, server, electronic device, computer-readable storage medium, computer program product or chip are all used to execute the corresponding methods provided above. Therefore, the beneficial effects they can achieve can be referred to the beneficial effects in the corresponding methods provided above, and will not be repeated here.

[0278] Through the above description of the embodiments, those skilled in the art will understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0279] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0280] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A data crawling method, characterized in that, The method is applied to a data crawling system, which includes a web client and a server, and the method includes: The web client displays a second page, which is used to configure different storage media; The web client receives a second configuration operation for the different storage media through the second page, and sends the configuration content for the different storage media to the server storage. The second configuration operation includes at least connection configuration for the different storage media. The web client displays a first page, which is used to configure the data crawling task; The web client receives a first configuration operation for the data crawling task through the first page, and sends the configuration content of the data crawling task to the server for storage. The first configuration operation includes at least configuring the data source, crawling time interval and storage medium of the data crawling task. The storage medium is used to store the crawled web page data. The configuration of the storage medium of the data crawling task includes selecting a storage medium from the different storage media configured in the second configuration operation. When the web client receives an operation to start the data crawling task, it sends a task request to the server, requesting the server to execute the data crawling task. The task request carries the identifier of the data crawling task. The server crawls web page data from the URL corresponding to the crawling data source and saves the web page data to the configured first storage medium; The server crawls webpage data from the URL corresponding to the crawling data source, including: The server obtains the crawling data source corresponding to the data crawling task based on the identifier of the data crawling task, and stores the crawling data source into a first queue according to the crawling trigger time, wherein the crawling trigger time is determined by the server based on the current system time and the crawling time interval. The server obtains the crawling data source from the first queue, and crawls web page data according to the URL corresponding to the crawling data source when the current system time reaches the crawling trigger time; If the current system time has not reached the crawling trigger time, the server calculates the waiting time based on the crawling trigger time and the current system time; If the waiting time is greater than the first preset duration, the server executes a local lock for the first preset duration, releases the local lock, and then retrieves the crawling data source from the first queue again. If the waiting time is not greater than a first preset duration, the server executes a local lock on the waiting time, releases the local lock, and then retrieves the crawling data source from the first queue again.

2. The method according to claim 1, characterized in that, The first page includes a first control, and the web client receives a first configuration operation for the data crawling task through the first page, including: In response to the user's click on the first control, the web client displays a first input control for configuring the data crawling task; The system receives the crawling data source, the crawling time interval, and the storage medium input by the user on the first input control, and completes the first configuration operation.

3. The method according to claim 2, characterized in that, The method by which the user inputs the storage medium on the first input control includes: selecting at least one storage medium from a list of created storage media.

4. The method according to claim 1, characterized in that, The second page includes a second control, through which the web client receives a second configuration operation on the storage medium, including: In response to the user's click on the second control, the web client displays a third page for configuring the connection to the storage medium; Receive the storage source selection operation input by the user on the third page, and display a second input control configured according to the selected storage source; The system receives connection configuration information for the storage medium input by the user on the second input control and completes the second configuration operation. The connection configuration information includes information on creating data tables or indexes based on the selected storage source.

5. The method according to claim 4, characterized in that, The method further includes: The web client displays a fourth page, which is used to configure the storage source. The storage source is selected by the user when the user enters the storage source selection operation on the third page. The third configuration operation for the storage source is received through the fourth page, and the third configuration operation includes at least the configuration of the hostname of the storage source.

6. The method according to claim 1, characterized in that, After the web client receives the first configuration operation for the data crawling task through the first page, the method further includes: The web client displays the configuration content of the data crawling task and a third control on the first page. The third control is used to trigger the start and stop of the data crawling task.

7. The method according to claim 6, characterized in that, The web client receives an operation to start the data crawling task, including: The web client receives a user's click on the third control, triggering the start of the data crawling task.

8. The method according to any one of claims 1 to 7, characterized in that, Saving the webpage data to the configured first storage medium includes: The server obtains information about the first storage medium corresponding to the data crawling task based on the identifier of the data crawling task. If the information of the first storage medium is not empty, the server obtains the connection configuration information corresponding to the first storage medium; Based on the connection configuration information, the webpage data is saved to the first storage medium.

9. The method according to claim 8, characterized in that, The step of saving the webpage data to the first storage medium according to the connection configuration information includes: The server determines the type of the first storage medium; If the connection configuration information is not empty, a connection client corresponding to the first storage medium is created according to the type of the first storage medium and the connection configuration information. According to the connected client, the webpage data is saved to the first storage medium.

10. The method according to claim 9, characterized in that, After creating the connection client corresponding to the first storage medium, the method further includes: The server stores information about the connected client.

11. The method according to any one of claims 1 to 7, characterized in that, The number of storage media configured for the data crawling task is greater than or equal to 1.

12. The method according to claim 11, characterized in that, The storage medium includes at least one of the following: MySQL type storage medium, Kafka type storage medium, and Elasticsearch type storage medium.

13. A data crawling system, characterized in that, The data crawling system includes a web client and a server, and the data crawling system is used to perform the method as described in any one of claims 1 to 12.

14. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes instructions that, when executed on the data crawling system, cause the data crawling system to perform the method as described in any one of claims 1 to 12.

Images