Distributed crawler method and system

Abstract

The application relates to the field of crawlers, in particular to a distributed crawler method and system, which improves the data crawling efficiency, data crawling scheduling capability and flexibility. The scheme comprises the following steps: collecting URL data, saving the crawled URL data into a URL data set, performing hash calculation on the collected URL, if the calculation result does not belong to the URL data set, adding the URL to a task node, otherwise discarding the URL; adopting a neural network to distill the URL, intercepting the URL exceeding a preset length, taking the distilled result as specific features; establishing a DQN model, inputting the specific features into the DQN model, dynamically distributing the crawling tasks according to the sub-node throughput, and performing data grabbing; and mining preference data in the data grabbing process through a knowledge graph. The application is suitable for data crawling.

Description

Technical Field

[0001] This invention relates to the field of web crawling, and more specifically to a distributed web crawling method and system. Background Technology

[0002] The emergence and development of distributed web crawlers are mainly due to the following factors:

[0003] 1. Cloud computing technology: Cloud computing architecture provides a distributed, highly available and scalable infrastructure that enables the deployment and scheduling of web crawlers.

[0004] 2. Distributed computing technology: Distributed computing architecture is based on clusters and parallel computing, which can effectively handle large-scale data and tasks and improve the efficiency of data capture and processing.

[0005] 3. Machine learning technology: Machine learning technology can automatically learn features and train classification models, which can help to automatically identify and classify target websites.

[0006] 4. Big Data Technology: Big data technology can assist in data processing, storage, and analysis, and can effectively support the data crawling and processing work of distributed web crawlers.

[0007] Existing web crawling technologies, such as the distributed web crawling system disclosed in CN109829094A, include: a URL reading and writing module, a URL crawling module, a document parsing module, and a persistence module. The URL reading and writing module, based on the MapReduce-based Map phase, reads URLs from the input stream and writes them to the output stream. The URL crawling module uses the URLs written to the output stream as access addresses and, according to a preset network access pattern, downloads the target documents corresponding to those access addresses based on the Map phase. The document parsing module, based on the Map phase, extracts target data from the target documents according to a preset method. The persistence module, based on the Map phase, stores the target data in the Hadoop Distributed File System according to a preset path and persistence rules. This solution modularizes the distributed web crawling system, enabling information exchange through data transfer between modules, thus improving the system's scalability, availability, maintainability, and scheduling capabilities.

[0008] However, this solution does not process the URL, resulting in low reading efficiency. It only relies on data transmission between modules to improve scheduling capabilities, but this method has low scheduling capabilities. Furthermore, this solution is just a conventional distributed crawler, which is not flexible enough when crawling data. Summary of the Invention

[0009] The purpose of this invention is to provide a distributed web crawler method and system that improves the efficiency, scheduling capability, and flexibility of data crawling.

[0010] This invention achieves the above objectives by adopting the following technical solution: a distributed web crawler method, comprising:

[0011] Step 1: Collect URL data and save the crawled URL data to the URL data set. Perform hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, add the URL to the task node; otherwise, discard the URL.

[0012] Step 2: Use a neural network to distill the URLs, truncate URLs that exceed the preset length, and use the distilled results as specific features;

[0013] Step 3: Establish a DQN model, input the specific features into the DQN model, dynamically allocate crawling tasks according to the throughput of child nodes, and perform data crawling.

[0014] Step 4: Mining preferred data during the data crawling process using knowledge graphs, including identifying entities in the crawled text, linking the identified entities with entities in the knowledge base, labeling entities with a unified ID, extracting relationships between entities from the crawled text based on entity links, building connections between URLs and entities during crawling, selecting preferred entities, and sorting the URLs to be crawled so that URLs associated with the target entities have higher priority.

[0015] Furthermore, the method also includes: Step 5, storing the data crawled by the child nodes into the target database, using a relational database for storage, using hash calculation to deduplicate the target text data, and then putting the data into the front end for display.

[0016] Furthermore, in step 1, the method also includes: adding a crawler proxy pool, randomly selecting a proxy node from the proxy pool for each crawl, and adding latency limits to the proxy servers in the proxy pool.

[0017] Furthermore, in step 2, an RNN model is used to distill the URL. Specifically, this includes adding a hidden layer to the basic RNN structure to distill the URL features. The hidden layer is a fully connected layer that outputs a vector of dimension 1 to extract URL features.

[0018] Furthermore, the system load and task parameters input to the DQN model are first given, including CPU utilization, memory utilization, queue task length, task processing latency, and URL distillation results.

[0019] Then, the DQN model is used for unsupervised learning of the target data, and an experience pool is used to store multiple experiences. The DQN model includes a current value network and a target value network. The current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is calculated as: targetQ = R + γ*max(s', a * ;θ), γ represents the learning rate;

[0020] The current value network is calculated as follows: predictQ = Q(s, a; θ), where s is the current state, a is the action, s' is the next state, and a' is the next action. * For the selected action, R represents the previous network parameters; the loss is calculated using the following formula:

[0021] The parameters of the current value network are updated through gradient backpropagation of the neural network. Specifically, the process includes: initializing the DQN model, inputting the current target child node state, obtaining all action Q values ​​through DQN, selecting an action to input into the environment, obtaining a new state and reward, then calculating the target value network, calculating the loss, updating the network, and finally adding the action state (s,a,r,s') to the memory unit, where r represents the reward.

[0022] A distributed web crawler system, the system comprising:

[0023] The data acquisition module is used to collect URL data and save the crawled URL data to the URL data set. It performs hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, the URL is added to the task node; otherwise, the URL is discarded.

[0024] The data distillation module is used to distill URLs using a neural network, truncate URLs that exceed a preset length, and use the distilled result as a specific feature.

[0025] The data crawling module is used to build a DQN model, input the specific features into the DQN model, dynamically allocate crawling tasks according to the throughput of child nodes, and perform data crawling.

[0026] The data mining module is used to mine preferred data during the data crawling process using knowledge graphs. This includes identifying entities in the crawled text, linking the identified entities with entities in the knowledge base, labeling entities with a unified ID, extracting relationships between entities from the crawled text based on entity links, building connections between URLs and entities during crawling, selecting preferred entities, and sorting the URLs to be crawled so that URLs associated with the target entities have higher priority.

[0027] Furthermore, the system also includes a data deduplication module, which is used to store the data crawled by the child nodes into the target database using a relational database, and to use hash calculation to deduplicat the target text data before putting the data into the front end for display.

[0028] Furthermore, the data acquisition module is also used to increase the crawler proxy pool. Each crawl randomly selects a proxy node from the proxy pool for crawling, and adds latency limits to the proxy servers in the proxy pool.

[0029] Furthermore, the data distillation module is specifically used to distill URLs using an RNN model. Specifically, it involves adding a hidden layer to the basic RNN structure to distill URL features. The hidden layer is a fully connected layer that outputs a vector of dimension 1 to extract URL features.

[0030] Furthermore, the data capture module is specifically used to first provide the system load and task parameters input to the DQN model, including CPU utilization, memory utilization, queue task length, task processing latency, and URL distillation results.

[0031] Then, the DQN model is used for unsupervised learning of the target data, and an experience pool is used to store multiple experiences. The DQN model includes a current value network and a target value network. The current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is calculated as: targetQ = R + γ*max(s', a * ;θ), γ represents the learning rate;

[0032] The current value network is calculated as follows: predictQ = Q(s, a; θ), where s is the current state, a is the action, s' is the next state, and a' is the next action. * For the selected action, R represents the previous network parameters; the loss is calculated using the following formula:

[0033] The parameters of the current value network are updated through gradient backpropagation of the neural network. Specifically, the process includes: initializing the DQN model, inputting the current target child node state, obtaining all action Q values ​​through DQN, selecting an action to input into the environment, obtaining a new state and reward, then calculating the target value network, calculating the loss, updating the network, and finally adding the action state (s,a,r,s') to the memory unit, where r represents the reward.

[0034] The beneficial effects of this invention are as follows:

[0035] This invention utilizes URL distillation technology to make URL features easier to extract and incorporate them as feature vectors into a scheduling management model. This allows child nodes that are more likely to crawl the target URL to receive the task, thereby improving crawling efficiency. The DQN model is used for scheduling and management of crawling tasks, which improves scheduling and management capabilities. The preference data mining based on knowledge graphs enhances the crawling of target preferences, making it easier for crawlers to prioritize crawling and greatly improving the flexibility of data crawling. Attached Figure Description

[0036] Figure 1 This is a flowchart of a distributed crawler method provided in an embodiment of the present invention;

[0037] Figure 2 A schematic diagram of the RNN model structure provided in an embodiment of the present invention;

[0038] Figure 3 This is a schematic diagram of the hidden layer structure in the RNN model provided in an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram of the DQN model structure provided in an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of a one-to-many storage relationship provided in an embodiment of the present invention. Detailed Implementation

[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0042] This invention provides a distributed web crawler method, such as... Figure 1 As shown, it includes:

[0043] S1, Data Acquisition

[0044] Collect URL data and save the crawled URL data to a URL data set. Perform a hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, add the URL to the task node; otherwise, discard the URL. The hash calculation is as follows: T = Hash(URL), where T is the calculation result.

[0045] Simultaneously, a crawler proxy pool is added. Each crawl randomly selects a proxy node from the pool for crawling. Latency limits are imposed on the proxy servers in the pool to prevent them from being blocked by the target server.

[0046] S2, Data Distillation

[0047] Data distillation, also known as URL distillation, is used because target websites may block some web crawlers. URL distillation ensures that similar URLs are crawled by nodes that are easier to access. A neural network is used to distill the URLs, truncating features from excessively long URLs. The distilled results are then input as specific features into the DQN network. The URL distillation model is trained using an RNN model. Since URL length has less impact on the learning outcome, longer URLs have a smaller influence on the learning results, making RNNs suitable for distillation.

[0048] The RNN model used is as follows: Figure 2 As shown, a hidden layer is added to the basic RNN structure to distill URL features. The URL distillation model during training, (x1,x2,x3,...xn), represents the URL sequence. The model input length is 512, and the output is 512.

[0049] Hidden layer structure such as Figure 3 As shown, logvar and mu are fully connected layers, and their output is a 1-dimensional vector used to extract URL features. First, logvar is used to obtain the calculation factor std, and the calculation process is as follows: std = e logvar / 2 The final hidden layer calculation result is z, which is calculated as follows: z = randn(std.size()) * std + mu. During training, the URL length is first truncated. Assume the training data URL is X = [x1, x2, x3, ..., x n ], X * =[x1,x2,x3,...,x 512 URLs longer than 512 characters are truncated; those shorter than 512 characters are padded with null characters ('\0'). The training label is the training URL, represented as Y = [y1, y2, y3, ..., y...]. 512 Therefore, the propagation loss is: loss(x) i ,y i )=-w[y i logx i +(1-y i log(1-x) i [], where w is a hyperparameter and can be left unset. During prediction, the distilled result z is used as an input dimension in subsequent calculations.

[0050] S3, Data Crawling

[0051] A DQN model is established, and the specific features are input into the DQN model. Crawling tasks are dynamically allocated according to the throughput of child nodes to perform data crawling.

[0052] Distributing URLs to different crawler programs enables distributed crawling. Specifically, this involves using a DQN (Dynamic Queuing) model for scheduling. The DQN model is established to dynamically allocate crawling tasks based on effective information such as child node throughput. The system load and task parameters for the model input are given in the table below.

[0053]

[0054] The DQN model is used for unsupervised learning of the target data. An experience pool is used to store multiple experiences (s, a, r, s'), where s represents the state (as shown in the parameter list above), a represents the action (i.e., selecting a crawler node), r represents the value (calculated by handling latency), and s' represents the next state. The value r is calculated as follows: assuming a latency threshold of ε, when assigning the target child node i, the value r is: r = log(t... i -ε).

[0055] DQN model such as Figure 4 As shown in the figure, the current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is the same as the current value network, except that the parameters of the target value network are the previous parameters of the current value network.

[0056] The target value network is used to predict the actual Q value. The target value network is calculated as: targetQ = R + γ*max(s', a) * ;θ), γ represents the learning rate;

[0057] The current value network is calculated as: predictQ = Q(s, a; θ), which represents obtaining the Q value of choosing action a in state s, where s is the current state, a is the action, s' is the next state, and a' is the action. * For the selected action, R represents the previous network parameters; the loss is calculated using the following formula:

[0058] All parameters of the current value network are updated using gradient backpropagation in the neural network. Furthermore, every N time steps, all parameters of the current value network are copied to the target value network.

[0059] The specific process includes:

[0060] 1. Initialize DQN, input the current target child node state, and obtain the Q values ​​of all actions through DQN.

[0061] 2. Select an action (select a crawler node) and input it into the environment to obtain a new state and reward r.

[0062] 3. Calculate the target value network.

[0063] 4. Calculate the loss.

[0064] 5. Update the network.

[0065] 6. Add the action state (s, a, r, s') to the memory unit.

[0066] S4, Data Mining

[0067] By using knowledge graphs to mine biased data during the data crawling process, the crawler becomes more biased, specifically including:

[0068] 1. Entity Recognition: Identifying entities in text. This is typically achieved using natural language processing techniques, such as natural language segmentation, part-of-speech tagging, and named entity recognition. Here, Stanford CoreNLP is used to perform entity recognition on the crawled content.

[0069] 2. Entity Linking: Linking identified entities with entities in the knowledge base and assigning each entity a unique ID. Entity linking is implemented using Solr.

[0070] 3. Relation Extraction: Building upon entity links, relation extraction extracts the relationships between entities from the text. Relation extraction requires graph analysis and natural language processing (NLP) techniques. Graph analysis processes the relationships between entities and transforms them into a graph form. NLP techniques are used for semantic analysis, syntactic analysis, and syntax analysis of the text. OpenNRE is used to build the relation model.

[0071] After the knowledge graph is built, it is displayed on the front end. Simultaneously, during the crawling process, relationships are established between URLs and knowledge network entities, stored in key-value pairs (entity relationship - URL), forming a one-to-many relationship, such as... Figure 5 As shown.

[0072] Finally, by selecting preferred entities, the URLs to be crawled are sorted, so that the URLs associated with the target entities are given higher priority.

[0073] In one embodiment of the present invention, the distributed crawler method further includes S5: storage and analysis, whereby the data crawled by the child nodes is stored in the target database using a relational database, and the target text data is deduplicated using a hash method. Finally, the data is displayed on the front end.

[0074] The present invention also provides a distributed crawler system, the system comprising:

[0075] The data acquisition module is used to collect URL data and save the crawled URL data to the URL data set. It performs hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, the URL is added to the task node; otherwise, the URL is discarded.

[0076] The data distillation module is used to distill URLs using a neural network, truncate URLs that exceed a preset length, and use the distilled result as a specific feature.

[0077] The data crawling module is used to build a DQN model, input the specific features into the DQN model, dynamically allocate crawling tasks according to the throughput of child nodes, and perform data crawling.

[0078] The data mining module is used to mine preferred data during the data crawling process using knowledge graphs. This includes identifying entities in the crawled text, linking the identified entities with entities in the knowledge base, labeling entities with a unified ID, extracting relationships between entities from the crawled text based on entity links, building connections between URLs and entities during crawling, selecting preferred entities, and sorting the URLs to be crawled so that URLs associated with the target entities have higher priority.

[0079] In one embodiment of the present invention, the system further includes a data deduplication module, which is used to store the data crawled by the child nodes into the target database, using a relational database for storage, and using a hash method to deduplicate the target text data before putting the data into the front end for display.

[0080] In one embodiment of the present invention, the data acquisition module is further used to add a crawler proxy pool, randomly select a proxy node from the proxy pool for each crawl, and add latency limits to the proxy servers in the proxy pool.

[0081] In one embodiment of the present invention, the data distillation module is specifically used to distill URLs using an RNN model, specifically including: adding a hidden layer to the basic RNN structure to distill URL features. The hidden layer is a fully connected layer that outputs a vector of dimension 1 to extract URL features.

[0082] In one embodiment of the present invention, the data capture module is specifically used to first provide the system load and task parameters input to the DQN model, wherein the task parameters include CPU utilization, memory utilization, queue task length, task processing latency, and URL distillation results;

[0083] Then, the DQN model is used for unsupervised learning of the target data, and an experience pool is used to store multiple experiences. The DQN model includes a current value network and a target value network. The current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is calculated as: targetQ = R + γ*max(s', a * ;θ), γ represents the learning rate;

[0084] The current value network is calculated as follows: predictQ = Q(s, a; θ), where s is the current state, a is the action, s' is the next state, and a' is the next action. * For the selected action, R represents the previous network parameters; the loss is calculated using the following formula:

[0085] The parameters of the current value network are updated through gradient backpropagation of the neural network. Specifically, the process includes: initializing the DQN model, inputting the current target child node state, obtaining all action Q values ​​through DQN, selecting an action to input into the environment, obtaining a new state and reward, then calculating the target value network, calculating the loss, updating the network, and finally adding the action state (s,a,r,s') to the memory unit, where r represents the reward.

[0086] In summary, this invention greatly improves the efficiency, scheduling capability, and flexibility of data crawling.

Claims

1. A distributed web crawling method, characterized in that, include: Step 1: Collect URL data and save the crawled URL data to the URL data set. Perform hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, add the URL to the task node; otherwise, discard the URL. Step 2: Use a neural network to distill the URLs, truncate URLs that exceed the preset length, and use the distilled results as specific features; Specifically, the URL is distilled using an RNN model, which includes adding a hidden layer to the basic RNN structure to distill the URL features. The hidden layer is a fully connected layer that outputs a vector of dimension 1 to extract URL features. Step 3: Establish a DQN model, input the specific features into the DQN model, and dynamically allocate crawling tasks based on the throughput of child nodes to perform data crawling; the dynamic allocation of crawling tasks based on the throughput of child nodes specifically includes: First, the system load and task parameters input to the DQN model are given. The task parameters include CPU utilization, memory utilization, queue task length, task processing latency, and URL distillation results. Then, the DQN model is used for unsupervised learning of the target data, and an experience pool is used to store multiple experiences. The DQN model includes a current value network and a target value network. The current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is calculated as follows: , Indicates the learning rate; The current value network is calculated as follows: s represents the current state, and a represents the action. For the next state, For the selected action, R represents the previous network parameters; the loss is calculated using the following formula: ; The current value network's parameters are updated using gradient backpropagation through a neural network. Specifically, this involves: initializing the DQN model, inputting the current target child node state, obtaining the Q-values ​​of all actions via DQN, selecting an action as input to the environment, obtaining the new state and reward, then calculating the target value network, calculating the loss, updating the network, and finally updating the action state. Added to the memory unit, where r represents the reward; Step 4: Mining preferred data during the data crawling process using knowledge graphs, including identifying entities in the crawled text, linking the identified entities with entities in the knowledge base, labeling entities with a unified ID, extracting relationships between entities from the crawled text based on entity links, building connections between URLs and entities during crawling, selecting preferred entities, and sorting the URLs to be crawled so that URLs associated with the target entities have higher priority.

2. The distributed crawler method according to claim 1, characterized in that, The method also includes: Step 5, storing the data crawled by the child nodes into the target database, using a relational database for storage, using hash calculation to remove duplicates from the target text data, and then putting the data into the front end for display.

3. The distributed crawler method according to claim 1, characterized in that, In step 1, the method also includes: adding a crawler proxy pool, randomly selecting a proxy node from the proxy pool each time for crawling, and adding latency limits to the proxy servers in the proxy pool.

4. A distributed crawler system, used to implement the distributed crawler method as described in any one of claims 1-3, characterized in that, The system includes: The data acquisition module is used to collect URL data and save the crawled URL data to the URL data set. It performs hash calculation on the collected URLs. If the calculation result does not belong to the URL data set, the URL is added to the task node; otherwise, the URL is discarded. The data distillation module is used to distill URLs using a neural network, truncating URLs that exceed a preset length and using the distilled result as specific features. Specifically, the data distillation module uses an RNN model to distill URLs, which includes adding a hidden layer to the basic RNN structure to distill URL features. The hidden layer is a fully connected layer that outputs a vector of dimension 1 to extract URL features. The data crawling module is used to build a DQN model, input the specific features into the DQN model, dynamically allocate crawling tasks according to the throughput of child nodes, and perform data crawling. Specifically, the data crawling module is used to first provide the system load and task parameters input to the DQN model. The task parameters include CPU utilization, memory utilization, queue task length, task processing latency, and URL distillation results. Then, the DQN model is used for unsupervised learning of the target data, and an experience pool is used to store multiple experiences. The DQN model includes a current value network and a target value network. The current value network is used to predict the estimated Q value, and the target value network is used to predict the actual Q value. The target value network is calculated as follows: , Indicates the learning rate; The current value network is calculated as follows: s represents the current state, and a represents the action. For the next state, For the selected action, R represents the previous network parameters; the loss is calculated using the following formula: ; The current value network's parameters are updated using gradient backpropagation through a neural network. Specifically, this involves: initializing the DQN model, inputting the current target child node state, obtaining the Q-values ​​of all actions via DQN, selecting an action as input to the environment, obtaining the new state and reward, then calculating the target value network, calculating the loss, updating the network, and finally updating the action state. Added to the memory unit, where r represents the reward; The data mining module is used to mine preferred data during the data crawling process using knowledge graphs. This includes identifying entities in the crawled text, linking the identified entities with entities in the knowledge base, labeling entities with a unified ID, extracting relationships between entities from the crawled text based on entity links, building connections between URLs and entities during crawling, selecting preferred entities, and sorting the URLs to be crawled so that URLs associated with the target entities have higher priority.

5. The distributed crawler system according to claim 4, characterized in that, The system also includes a data deduplication module, which stores the data crawled by the child nodes into the target database using a relational database, performs hash calculations to deduplicate the target text data, and then puts the data into the front end for display.

6. The distributed crawler system according to claim 4, characterized in that, The data acquisition module is also used to add a crawler proxy pool. Each time, a proxy node is randomly selected from the proxy pool for crawling, and latency limits are added to the proxy servers in the proxy pool.

Images