Web spider theme search system based on improved cloud platform

Abstract

The web spider thematic search system based on the improved cloud platform provided by the present invention improves the HITS algorithm, a web analysis algorithm based on a link structure, and the web analysis algorithm based on the topic similarity calculation of a VSM vector space model, proposes an improved web spider model algorithm, proposes an overall framework model based on the Hadoop cloud platform web spider, designs and implements the storage structure of the cloud platform web spider on the file system HDFS, and implements the MapReduce algorithm for each functional module based on module division; proposes an improved task allocation algorithm, which can take into account the uniform allocation and the load of each crawling sub-node, and improves the crawling efficiency and accuracy of the cloud platform web spider system; the results show that the cloud platform web spider system based on Hadoop proposed and implemented by the present invention is feasible and effective, can greatly improve the accuracy and efficiency of thematic search, and can retrieve topic-related information comprehensively, quickly and accurately.

Description

Technical Field

[0001] The present invention relates to a web spider theme search system, in particular to a web spider theme search system based on an improved cloud platform, belonging to the technical field of theme search systems. Background Art

[0002] With the increasing popularity and rapid development of Internet technology, facing such a huge and diverse amount of information data, general search engines as the main means of obtaining information can no longer meet people's retrieval needs for information related to specific fields or topics. In view of this, thematic search engines and web spider technologies for specific fields or topics have been widely used, but as the amount of network information grows exponentially, traditional web spiders that rely only on a single computer for crawling can no longer meet the performance and scalability requirements of thematic search in the big data environment. The Hadoop cloud platform computing framework can solve this problem well.

[0003] In recent years, the research and development of topic-based search engines has become a hot topic. The specialization, personalization and strong pertinence of topic-based search engines have made them a hot topic and trend in the current search engine field. Some topic-based search engines have emerged in the prior art. Youdao is a search engine developed by NetEase after ending its cooperation with Google. It provides searches for different topics such as videos, web pages, music, and dictionaries. The most distinctive one is the dictionary-based topic search, which provides English-Chinese translation of keywords entered by users, basic definitions, online definitions, a large number of bilingual examples, and standard pronunciation reading. Scirus is the world's most comprehensive technology search engine specifically used for technology information retrieval. It can search both websites and journal resources, and focuses on technology content. Scirus' basic search is similar to that of general search engines, and supports operations such as AND or NOT. Advanced search can be searched by different fields. On the other hand, as the amount of network information grows exponentially, traditional web spiders that rely solely on a single computer for crawling can no longer adapt to the performance and scalability requirements of topic-based search in a big data environment.

[0004] In addition, the open source spiders in the prior art, such as Larbin, are a single spider with a simple design that only crawls web pages. Users can configure some crawling parameters by themselves, and the crawling performance is good, but it pays no attention to page analysis or data storage, etc., which can easily cause deduplication misjudgment; WebLech is a Web site download and mirroring tool that imitates the behavior of a Web browser as standardly as possible and uses multi-threading to download Web sites according to functional requirements; the open source product Heritrix uses a method of deep traversal of web pages to completely crawl non-text content without making any judgment, but the spider has many control parameters, is more complicated to configure, and does not support Chinese well enough.

[0005] With the further development of the Internet, the computing power of cloud platforms for massive data needs to be further improved. After Google publicly released its product architecture, its cloud platform file system GFS, and related research on the high-performance cloud platform computing model MapReduce, the Nutch search engine proposed its own cloud platform file system NDFS after analyzing GFS, and then introduced the MapReduce cloud platform computing model into Nutch development. Hadoop is a framework removed from the Nutch project that mainly uses the MapReduce cloud platform computing model for large-scale computing. The introduction of the Hadoop cloud platform framework makes the processing of massive data and the implementation of cloud platform computing convenient and efficient.

[0006] In summary, the shortcomings of the existing topic search system mainly include:

[0007] First, in the face of such a huge and diverse amount of information data on the Internet, general search engines as the main means of obtaining information can no longer meet people's retrieval needs for information related to specific fields or topics. Thematic search engines and web spider technologies for specific fields or topics have been widely used, but as the amount of network information grows exponentially, traditional web spiders that rely only on a single computer for crawling can no longer meet the performance and scalability requirements of thematic search in a big data environment. Existing technologies cannot solve this problem well.

[0008] Second, some of the existing open source spiders are single spiders with simple design that only crawls web pages. Users can configure some crawling parameters by themselves. They neither pay attention to page analysis nor data storage, which can easily cause deduplication misjudgment. Some use the method of deep traversal of web pages to completely crawl non-text content without judgment. The spider has many control parameters, which is troublesome to configure and does not support Chinese well enough.

[0009] Third, the types of information crawled by the search system of the prior art are not targeted. In terms of data acquisition, the types of information crawled cannot be screened based on the topics provided by users; the cross-platform performance is poor, and the software cannot be transplanted based on the requirements of different platform architectures, or in the process of software transplantation, a lot of changes need to be made to meet the needs of users running under different platform architectures; the cost is high and the performance is low. The prior art has high requirements for the configuration of the search system computer. It is necessary to crawl the web page information containing massive data on the Internet under high cost conditions, and the efficiency of processing the crawled web page information is low. There is a lack of excellent crawling and task allocation algorithms, and it is impossible to achieve comprehensive, fast and accurate retrieval of subject-related information;

[0010] Fourth, the existing technology has poor robustness and system stability. On the one hand, when crawling web page information, the system continuously establishes network connections with a huge number of servers to obtain web page information data. In the process, the system not only needs to parse the web page information in a timely manner, but also needs to promptly handle those URL links that cannot be linked or web server abnormalities or failures that cause failure to connect. The existing technology cannot guarantee that the system can continue to operate normally; on the other hand, the system operates in an environment built by multiple computer nodes. When the cluster works continuously and uninterruptedly, various unpredictable abnormalities or failures will inevitably occur in each computer node. When a computer node has an unpredictable abnormality or failure and cannot operate normally, the system cannot timely discover and locate the abnormal or failed computer node, and cannot allocate the computing tasks of the failed node to other normally operating computer nodes. The system has poor robustness and system stability;

[0011] Fifth, the search system crawling in the prior art has poor manageability and cannot obtain detailed information such as the system operating status and the memory utilization rate, CPU utilization rate, etc. of each computer node in real time; the configurability is poor and the operating parameters cannot be dynamically adjusted in real time based on the system operating status; the scalability is poor and the computer nodes in the computer cluster cannot be dynamically added or deleted during system operation; the load balancing capability is weak and there is a lack of load balancing mechanism within the system. The task load of each computer node in the search system is unevenly distributed, and the crawling efficiency of the system fluctuates greatly. Summary of the invention

[0012] With the increasing popularity and rapid development of Internet technology, facing such a huge and diverse information data, general search engines as the main means of obtaining information can no longer meet people's retrieval needs for specific fields or subject-related information. In view of this, thematic search engines and web spider technologies for specific fields or subjects have been widely used, but as the amount of network information grows exponentially, traditional web spiders that only rely on a single computer to crawl can no longer adapt to the performance and scalability requirements of thematic search under a big data environment, and the Hadoop cloud platform computing framework can solve this problem well. Therefore, the present invention proposes an improved web spider algorithm and task allocation algorithm, and implements a cloud platform web spider system based on Hadoop, which can retrieve subject-related information comprehensively, quickly and accurately.

[0013] In order to achieve the above technical effects, the technical solution adopted by the present invention is:

[0014] The web spider topic search system based on the improved cloud platform includes: first, improving the HITS algorithm based on the link structure and the web analysis algorithm based on the VSM vector space model for topic similarity calculation, proposing an improved web spider model algorithm to evaluate the comprehensive value of web page information; second, in the process of implementing the cloud platform web spider on the cloud platform, proposing an improved task allocation algorithm, taking into account the uniform distribution and the load of each crawling sub-node, and improving the crawling rate and accuracy of the cloud platform web spider system; third, proposing an overall framework model of the cloud platform web spider based on Hadoop, designing and implementing the storage structure of the cloud platform web spider on the file system HDFS, and implementing the MapReduce algorithm for each functional module based on module division; fourth, implementing the cloud platform web spider system based on Hadoop and testing it;

[0015] The basic execution process of the cloud platform web spider system of the present invention is:

[0016] First, the user selects some high-quality initial URL seed sets based on the related subject information they want to obtain and puts them into the seed URL file. The initial URL seed set serves as the starting point for the system to add search, and the system selects URL links in the seed URL file to crawl web page information;

[0017] Second, after obtaining the corresponding URL link, the system establishes a network connection with the Web server corresponding to the URL link. If the network connection fails to be established and the waiting time exceeds a certain time, the system abandons the network connection and marks the URL link, and selects the next URL link from the URL link queue for access;

[0018] Third, if network communication with the Web server is successfully established, the system uses the MapReduce cloud platform computing model to crawl the information in the web page based on the http protocol, and stores the crawled information in the file system HDFS;

[0019] Fourth, after the webpage crawling is completed, the system further analyzes the webpage information and stores the parsed webpage content information in the parsed webpage library in the file system HDFS;

[0020] Fifth, parse the URL links contained in the web page and perform a deduplication and filtering operation on the links;

[0021] Sixth, store the URL links after link deduplication filtering in the link outbound URL library in the file system HDFS for future crawling work;

[0022] Seventh, if the conditions for the web spider to stop are not met, the system will evaluate the comprehensive value of each URL link according to the improved web spider model, and select a URL link with the highest priority, that is, the web page most relevant to the specified topic, for the next crawling work;

[0023] The improved web spider model is mainly designed as follows:

[0024] First, the cloud platform web spider system is given a seed URL set, and then the URL segmentation module extracts the URL links in the seed URL set to perform URL segmentation operations, and then stores them in the un-crawled URL library in the cloud platform file system HDFS;

[0025] Second, the webpage crawling module reads the corresponding URL link from the uncrawled URL library and crawls it, and stores the crawled webpage information in the Web initial webpage library located in the cloud platform file system HDFS;

[0026] Third, the web page parsing module parses the URL links contained in the web page and stores them in the link URL library in the cloud platform file system HDFS, and stores the parsed web page content information in the parsed web page library in the cloud platform file system HDFS;

[0027] Fourth, the hyperlink evaluator reads the outbound URL library in the cloud platform file system HDFS, and calculates the Hub weight and Authority weight of each URL link based on the HITS algorithm. At the same time, it constructs a topic description matrix and obtains the vector representation of the topic from the topic description matrix and the topic web page set. It combines word frequency statistics and content structure information to represent the web page vector, and uses the cosine angle theorem to obtain the relevance value for the topic vector and the web page vector. The page relevance evaluator performs word segmentation processing on the parsed web page content information based on the associated topic feature word library and counts the feature topic word frequency, and then calculates the topic relevance of the web page content based on the VSM vector space model.

[0028] Fifth, the web spider comprehensive value evaluator adopts the improved web spider model algorithm to calculate the comprehensive value of each URL link based on the link value and content theme relevance value, compares the web links in the crawling queue, and determines the order in which the web spider will crawl the URLs next.

[0029] Based on the improved cloud platform web spider theme search system, further, the overall framework model of the cloud platform web spider system of the present invention is:

[0030] First, the JobTracker process is created and generated on the Master node. It divides the seed URL set and distributes the URL crawling task to each Slave node based on the running status of each Slave node. The JobTracker process is also responsible for real-time monitoring of the associated running status of the system. The JobTracker is deployed on a separate computer node in the cluster and does not participate in the specific crawling work.

[0031] Second, the JobTracker process distributes the segmented web page link URL to each TaskTracker process. Each TaskTracker process runs on each Slave node. After receiving the crawling subtask distributed to it by the JobTracker process, the TaskTracker process starts the corresponding Map task to perform the crawling of web page information. The corresponding Map task starts multiple threads to crawl web page information. After the Map task is completed, the crawled information is transmitted to the Reduce task as an intermediate result in the form of <link URL, content data> key-value pairs.

[0032] Third, after the Map task outputs the intermediate results, the TaskTracker process will start the Reduce task to complete the web page analysis, link deduplication filtering and merging operations, and store the link URL and web page content information obtained based on the web page content analysis in the link URL library and parsed web page library in the file system HDFS respectively. The TaskTracker process will also continue to send heartbeats to the JobTracker process through RPC to report on the resource usage and task running status on each node.

[0033] Based on the improved cloud platform web spider theme search system, the improved web spider model uses the hyperlink analysis algorithm HITS to calculate the hyperlink structure value of each candidate URL, and calculates the Hub weight value and Authority weight value for each URL link. These two values ​​will be used in the calculation of the comprehensive value of the web page. Based on the HITS algorithm, the URL link v i , the calculation of its Hub weight value and Authority weight value is shown in Formula 1 and Formula 2:

[0034]

[0035]

[0036] Among them, v i , v j ∈B represents v i Exists to v j Hyperlink, vj , v i ∈C represents v j Exists to v i Hyperlink, A[v i ]、H[v i ] indicates v i The Hub weight value and Authority weight value are calculated through continuous iteration. In the first step of the iterative calculation, each URL link is assigned an initial value:

[0037] A[V i ]=1,H[V i ]=1 Formula 3

[0038] Finally, after iterative calculation, A[v i ]、H[v i ] Normalized, the calculation formula is formula 4 and formula 5:

[0039]

[0040]

[0041] Among them, v i , v j ∈B represents v j Exists to v i Hyperlink, v j , v i ∈C represents v j Exists to v i Hyperlink.

[0042] Based on the web spider topic search system of the improved cloud platform, further, the improved web spider model of the present invention adopts the TF-IDF formula based on the VSM vector space model to calculate the topic relevance of the web page information content. According to the vector space model description, for a web page q, it is formally represented as a feature word weighted vector Q, and the calculation formula is Formula 6:

[0043] Q={q1,q2,…,q i ,…,q m ) Formula 6

[0044] Among them, q1 represents the feature word d i The weighted value in page Q is calculated by the TF-IDF formula. The weighted value is calculated as Formula 7, where n is the dimension of the feature word vector space of the topic:

[0045] q i =tf i *idf i Formula 7

[0046] Among them, tf i Indicates the characteristic word d i The frequency of the term in document Q, idf i Indicates the characteristic word d i The inverse document frequency of is calculated as formula 8:

[0047]

[0048] Among them, m i Indicates the characteristic words d that appear in the sample page set i M represents the total number of pages in the sample page set. The topic relevance of page q is calculated by calculating the weighted vector Q of the feature word of the page and the topic feature vector U + The inner product of two vectors gives the result, calculated as Equation 9:

[0049]

[0050] Where m represents the eigenvector U + dimension, sim(q) represents the topic relevance of page q. The larger its value is, the greater the probability that the web page information of page q is associated with the topic. The improved web spider model proposed in the present invention does not directly define page q as being associated or not, but calculates the probability of the web page information being associated with the topic, and then calculates the comprehensive value of the URL link based on this probability value.

[0051] Based on the improved cloud platform web spider thematic search system, the comprehensive value of web pages is further evaluated: the Shark-Search algorithm is improved on the basis of using the vector space model to calculate the topic relevance of web page information content, and the associated information of the link structure is added to it. The calculation formula is formula 10:

[0052]

[0053] Where sim(i) represents the topic relevance of web page i, fa(i) represents the set of all web pages linking to web page i, Inde(i) represents the number of backlinks of web page i, that is, the number of web pages in fa(i), the value range of variable q is between 0 and 1, and variable q adjusts the proportion of link structure and web page text, te_sim(i) is the similarity between web page content and topic, and the specific calculation is shown in formula 11:

[0054] te_sim(i)=Sim(Q_Text(i),Th)×p

[0055] +Sim(Q_Tit(i),Th)×(1-p) Formula 11

[0056] Q_Text(i) is the text content of web page i, Q_Tit(i) is the title information of web page i, Th represents the keyword set of the topic, and variable p is the same as q, its value range is between 0 and 1. Sim is calculated by using the TF-IDF algorithm based on the vector space model to calculate the feature vector of the document, and then the cosine theorem is used to calculate the similarity, as shown in Formula 9;

[0057] Based on the calculation of the value of the hyperlink structure and the calculation of the subject relevance of the content, the improved web spider model algorithm of the present invention comprehensively considers the link value of web page q and the subject relevance of the content, and provides a calculation formula for calculating the comprehensive value of the web page, which is Formula 12:

[0058] Value(i)=(H[i]+A[i])×(s+sim(i)) Formula 12

[0059] Among them, Value(i) is the comprehensive value of the web page q to be finally obtained, H[i] and A[i] represent the Hub weight value and Authority weight value of web page i respectively, which are the results calculated by using Formula 4 and Formula 5 based on the HITS algorithm, sim(i) represents the topic relevance of web page i, which is the result calculated by using Formula 10; in addition, a control factor s is added to Formula 12, and its value range is between 0 and 1. This design is because there is a special case that some pages with very high Hub weight values ​​and Authority weight values ​​may point to another set of topic pages, but their own web page content is not related to the topic. At this time, a control factor needs to be added to enable the web spider to crawl more topic-related pages.

[0060] Based on the improved cloud platform web spider theme search system, further, based on the overall framework model of the system, the uncrawled link library, initial web page library, parsed link library, and parsed web page library adopted by this system are proposed:

[0061] Uncrawled link library: The uncrawled URL library stores the currently pre-crawled URLs. After the URL segmentation module processes the URLs read from the URL seed file, they are stored in the file system HDFS. The web crawling module obtains URLs from the uncrawled URL library to perform specific web crawling work.

[0062] The initial webpage library stores the initial webpage information captured by the webpage capture module from each node in the cloud platform web spider. After processing, they are saved in the file system HDFS for use by subsequent webpage parsing modules.

[0063] The link analysis library stores the link analysis data obtained by the web page analysis module after the web page analysis module analyzes the initial web page information captured by each node in the cloud platform web spider. After processing, they are saved in the file system HDFS for subsequent crawling work;

[0064] The web page parsing library stores the page parsing data obtained by the web page parsing module from the initial web page information captured by each node in the cloud platform web spider. After processing, they are saved in the cloud platform file system HDFS for subsequent use;

[0065] The above four repositories are implemented on the Hadoop-based file system HDFS to be called by the functional modules in the cloud platform web spider system.

[0066] Based on the improved cloud platform, the web spider theme search system is further divided into three parts: spider sub-nodes, master node, and HDFS. The master node coordinates the crawling between spider sub-nodes, and the spider sub-nodes crawl pages, which are stored in HDFS.

[0067] The working mechanism of information and data interaction between the master node and the spider sub-node is:

[0068] First, the master control node knows how many spiders are deployed in the entire system based on its own configuration information, and starts the spiders by remote login from the master control node;

[0069] Second, after the spider child node is started, it sends a "Ready" status message to the main control node, indicating that the spider is ready to accept crawling tasks;

[0070] Third, after receiving the "Ready" message from the spider child node, the master control node generates an initial task, that is, encapsulates the URLs of some web pages to be crawled, and sends them to a certain path of the spider child node in the form of a file. Then, it sends a "Working" reply message to the spider node. If the master control node finds that there is no data in the current database, that is, there is no task to send, and there are still spiders crawling in the system, it will send a "Wait" reply message to the spider. If the spider system is started for the first time or all spiders are in the "Wait" state, it means that the user has not added a crawling task or the crawling is completed. The master control node will send a "Stop" message to each spider.

[0071] Fourth, after receiving the reply message of "Working", the spider child node goes to the corresponding path and loads the seed URL into the crawling queue. If it is the first start, it starts the crawling thread to start the crawling task. If it is not the first start, it continues to crawl the next task after the current crawling is completed. If the spider receives the "Wait" message, it sets the state to "Wait". After crawling the current task, it is in a waiting state until the master node wakes it up. If it receives the "Stop" message, the spider exits directly.

[0072] Fifth, in the process of crawling pages, when the extracted URLs reach a certain amount, they will be encapsulated into a data file, that is, written into a text file, and the spider will send it to the path of the master node and send the status information of "Dada" to the master node;

[0073] Sixth, after receiving the "Dada" message, the master control node loads the data into the database under the specified path;

[0074] Seventh, when the spider is about to complete the crawling task, it will also send a "Ready" status message to the main control node, and then return to the first interactive state.

[0075] Based on the improved cloud platform web spider theme search system, further, the improved task allocation algorithm: support cloud platform URL allocation through the secondary hash mapping algorithm, and make the system scale dynamically scalable, so that the system can perform efficient and balanced cloud platform URL allocation, and make the system have good dynamic configurability;

[0076] In a crawling cycle, the maximum number of logical nodes is logic_num. The number of currently running physical nodes is physi_num. Each crawling node has two tables: one is the physical node table, which is used to store the information of each node in the current system; the other is the logical node table, which stores the information of logic_num logical nodes. If the node corresponding to the node does not exist, the value is 0. After the URL is standardized, the URL is first hashed to the element in the corresponding logical table. If the element is not 0, the ID number is taken out to determine whether to route the URL to other crawling nodes. Otherwise, all URLs on the element are evenly distributed to the current physical nodes of the system through the second hash mapping. At this time, the ID number of the corresponding crawling node is obtained and it is determined whether to route the URL. In addition, in order to evenly distribute the URLs to each crawling node, a hash function with good performance needs to be selected. For this purpose, the present invention designs a URL hash function:

[0077]

[0078] In this function, in the first hash mapping, Q = logic_num, i.e., the number of logical nodes, and in the second hash mapping, Q = physic_num, i.e., the number of physical nodes. The Ascii function takes the ASCII code of each character in the string;

[0079] The present invention proposes an improved task allocation algorithm, which takes into account both uniform distribution and the load of each spider sub-node;

[0080] After the secondary hash mapping algorithm calculates the crawling sub-nodes to be assigned, the improved task splitting algorithm uses the weighted minimum connection scheduling algorithm to determine whether the load of the node allows the new URL crawling task to be assigned. Each sub-node uses a corresponding weight value to represent its processing performance. The default weight value is set to 1. The system administrator dynamically sets the weight value of the server. The weighted minimum connection scheduling makes the number of established connections of the server proportional to its weight value when scheduling new connections.

[0081] The algorithm flow of weighted minimum connection scheduling is as follows: Assume that there is a group of servers K = {K0, K1, ..., K n-1 ), J(K i ) indicates server K i The weight value, F(K i ) indicates server K i The current number of connections of all servers is F. sum =∑F(K i )(i=0,1,…,nl), the current new connection request will be sent to server K m , if and only if server K m The URL seed is sent only when the following conditions are met:

[0082]

[0083] Among them, J(K i ) is not 0, the log of the child node is fed back to the main control node at a certain interval, and the number of child server connections F(K i ) is obtained by reading the log, comparing the ratio of the number of child node connections and the prior weight value, obtaining the child node with the smallest load, and assigning a new crawling task.

[0084] Based on the improved cloud platform web spider theme search system, further, the modules of the cloud platform web spider system proposed by the present invention are divided into:

[0085] Module 1, URL segmentation module: Users select some high-quality initial URL seed sets based on the related topic information they want to obtain and put them into the seed URL file. The initial URL seed set is used as the starting point for the cloud platform web spider system to add searches, which is crucial for the system to crawl web pages of related topics. After obtaining the URL list, the URL segmentation module segments the seed URL set and assigns it to the TaskTracker process on each Slave node for crawling.

[0086] Module 2, web crawling module: based on the URL link obtained from the un-crawled URL library in the cloud platform file system HDFS and crawling web page information, first establish an HTTP connection with the web server where the URL is located, and then download the web page information and store it in the initial web page library in the cloud platform file system HDFS, waiting for the web page parsing module to process;

[0087] Module three, web page parsing module: after completing the crawling work, the web page crawling module saves the web page information to the initial web page library of the cloud platform file system. Then the web page parsing module distributes the tasks according to the number of blocks of input data, that is, the number of blocks of page content in the initial web page library of the file system HDFS. Then the web page parsing module uses the MapReduce cloud platform computing model to parse the web page content, including web page content information and link URL information. Finally, the web page parsing module parses the URL links contained in the web page and stores them in the link URL library in the file system HDFS, and stores the parsed web page content information in the parsed web page library in the file system HDFS.

[0088] Module 4, link filtering module: The link URLs obtained by the web page parsing module are filtered to remove duplicate links. For URL links that do not meet the standards, they will be normalized before they can be used for subsequent crawling work. For duplicate URL links, deduplication operations are required.

[0089] Module five, data storage module: The data storage module stores the parsed web page information and the URL links after filtering and deduplication in the file system HDFS for subsequent crawling work. After the cloud platform web spider system is started, it performs the tasks of URL segmentation, web page crawling, web page content parsing and link deduplication filtering function modules, and continues to run in a loop in parallel with the cloud platform until the cloud platform web spider system reaches the corresponding crawling end conditions.

[0090] Based on the improved cloud platform web spider theme search system, further, the URL segmentation module segments the URLs in the seed URL set, which is the first step for the cloud platform web spider system to officially start; first, the URL segmentation module obtains the seed URL from the seed URL set; then, after completing the segmentation task, it is distributed to each Slave node for the crawling work of the web crawling module. The spiders on each Slave node store the distributed URL set in their respective uncrawled URL library, and then select a URL web link from the uncrawled URL library as the start of a specific crawling task. The most important task of the system is to ensure that the crawling tasks on each node do not conflict. At this time, task allocation is particularly important. Therefore, after the URL segmentation module segments the seed URL set, the spiders on each TaskTracker each obtain the URL set assigned to them. The spiders on each TaskTracker cooperate with each other to perform crawling tasks in parallel. The work of the URL segmentation module is to segment the URL list into several fragments and save them in the uncrawled URL library in the file system HDFS for the web crawling module to adopt. The MapReduce model algorithm of the URL segmentation module is described as follows:

[0091] The first step is initialization: create a new MapReduce task and perform initialization operations on it;

[0092] The second step is preprocessing: preprocess the data through InputFormat and transform it into<key,value> The key-value pairs in this data format are convenient for MapReduce tasks to process;

[0093] The third step is data sharding: pre-processing the seed URL set, that is, obtaining URL link information from the seed URL set and sharding it;

[0094] Step 4: Map process: perform associated operations on each URL according to the improved task allocation algorithm;

[0095] Step 5, Combiner process: merge the values ​​of the same keyword key in the intermediate files output in step 4 to improve the processing efficiency of subsequent operation steps;

[0096] Step 6, Partitioner process: After the merging process in step 5, the intermediate results of the output of the Map task are split into several regions based on the key using a hash table.

[0097] Step 7, Reduce process: Based on the calculation results of the Map task, each Reduce task outputs the task allocation results of the URL links in the seed URL set;

[0098] Step 8. Store in HDFS: Process the output results of Reduce through OutputFormat, and store the URLs assigned to each crawling sub-node in the uncrawled link library in its file system HDFS for the next crawling work.

[0099] Compared with the prior art, the advantages and innovations of the present invention are:

[0100] First, in the context of the era of big data with massive growth of Internet information, the present invention proposes and completes a web spider thematic search system based on an improved cloud platform, improves the HITS algorithm based on the link structure and the web analysis algorithm based on the VSM vector space model for thematic similarity calculation, and proposes an improved web spider model algorithm, which evaluates the relevance of web page content and the subject from the aspect of web page content relevance, and evaluates the authority of the web page and the predictability of the related subject page from the aspect of link analysis, and can evaluate the comprehensive value of web page information; proposes an overall framework model of a web spider based on a Hadoop cloud platform, designs and implements the storage structure of the cloud platform web spider on the file system HDFS, and implements the MapReduce algorithm for each functional module based on module division; proposes an improved task allocation algorithm, which can take into account the uniform distribution and the load of each crawling sub-node, improves the system performance, optimizes the resource allocation, and improves the crawling efficiency and accuracy of the cloud platform web spider system; the results show that the cloud platform web spider system based on Hadoop proposed and implemented by the present invention is feasible and effective, can greatly improve the accuracy and efficiency of thematic search, and has very important research and development significance and strong application value;

[0101] Second, the web spider theme search system based on the improved cloud platform provided by the present invention has a strong pertinence in the type of information crawled. The cloud platform web spider system is based on the theme. In terms of data acquisition, the type of information crawled is information obtained by screening based on the theme provided by the user and calculating the theme relevance. It has good cross-platform performance. The cloud platform web spider system can perform software transplantation based on the requirements of different platform architectures. During the software transplantation process, the system only needs to make some basic parameter configuration changes to meet the needs of users running web spider systems under different platform architectures. The present invention proposes an improved web spider algorithm and task allocation algorithm, and realizes a cloud platform web spider system based on Hadoop, which can comprehensively, quickly and accurately retrieve theme-related information.

[0102] Third, the web spider theme search system based on the improved cloud platform provided by the present invention has the characteristics of low cost and high performance. The Hadoop cloud platform cluster environment adopted by the cloud platform web spider system can be built on multiple ordinary computers, and the configuration requirements of the computers are not high. The present invention can crawl web page information containing massive data on the Internet under low cost conditions, and can perform efficient cloud platform parallel processing on the crawled web page information;

[0103] Fourth, the robustness and system stability are good. On the one hand, the cloud platform web spider system continuously establishes network connections with a huge number of servers to obtain web information data during the crawling of web information. In the process, the system can not only parse the web information in a timely manner, but also promptly handle those URL links that cannot be linked or web server abnormalities or failures that cause the system to be unable to connect, thereby ensuring that the system can continue to operate normally; on the other hand, the system operates in a cloud platform computer cluster environment built by multiple computer nodes. Under the condition of long-term and uninterrupted operation of the cloud cluster, various unpredictable abnormalities or failures will inevitably occur in each computer node. When a computer node has an unpredictable abnormality or failure and cannot operate normally, the system can promptly discover and locate the abnormal or failed computer node, and assign the computing tasks of the failed node to other normally operating computer nodes to maintain the robustness and stability of the system;

[0104] Fifth, the web spider thematic search system based on the improved cloud platform provided by the present invention has good manageability and can obtain detailed information such as the system operation status and the memory utilization rate, CPU utilization rate, etc. of each computer node in real time; it has good configurability and can dynamically and in real time adjust certain operation parameters of the cloud platform web spider based on the system operation status, such as the number of threads enabled during crawling, the maximum crawling depth and other parameter configurations; it has good scalability and can dynamically add or delete each computer node in the computer cluster during system operation, has good scalability, and can greatly improve the crawling efficiency by increasing the computing nodes in the system; it has strong load balancing capability and can make full use of the advantages of parallel computing of the web spider system cloud platform to maximize the crawling efficiency, and designs a load balancing mechanism inside the cloud platform web spider system to reasonably and evenly distribute the task volume of each computer node, thereby greatly improving the crawling efficiency of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0105] Figure 1 It is a schematic diagram of the improved web spider model of the present invention.

[0106] Figure 2 It is the overall framework model diagram of the cloud platform web spider system proposed by the present invention.

[0107] Figure 3 It is a basic flow chart of the cloud platform web spider system of the present invention.

[0108] Figure 4 It is a cloud platform architecture diagram of the cloud platform web spider system of the present invention.

[0109] Figure 5 It is a schematic diagram of the module design of the cloud platform web spider system of the present invention.

[0110] Figure 6 It is a schematic diagram of the web page parsing process of the present invention. DETAILED DESCRIPTION

[0111] The following further describes the technical solution of the web spider thematic search system based on the improved cloud platform provided by the present invention in conjunction with the accompanying drawings, so that those skilled in the art can better understand the present invention and implement it.

[0112] The main work of the present invention includes: first, improving the HITS algorithm based on the link structure and the web page analysis algorithm based on the topic similarity calculation of the VSM vector space model, proposing an improved web spider model algorithm to evaluate the comprehensive value of web page information; second, in the process of implementing the cloud platform web spider on the cloud platform, proposing an improved task allocation algorithm, taking into account the uniform distribution and the load of each crawling sub-node, improving the system performance, optimizing resource allocation, and improving the crawling rate and accuracy of the cloud platform web spider system; third, proposing an overall framework model of a cloud platform web spider based on Hadoop, designing and implementing the storage structure of the cloud platform web spider on the file system HDFS, and implementing the MapReduce algorithm for each functional module based on module division; fourth, implementing the cloud platform web spider system based on Hadoop and testing it, the results show that the cloud platform web spider system based on Hadoop proposed and implemented by the present invention is feasible and effective, and can greatly improve the accuracy and efficiency of thematic search.

[0113] 1. Overall framework model of cloud platform web spider

[0114] The overall framework model of the cloud platform web spider system is designed as follows: first, the physical architecture is analyzed and designed, that is, the topological structure between each computer node in the Hadoop cloud platform computer cluster is designed and implemented. The present invention adopts a centralized topological structure, that is, one computer in the Hadoop cloud platform computer cluster is used as the Master node, and the other computers are used as each Slave node. The Master node is responsible for scheduling and managing all other Slave nodes. The Master node monitors the running status of each Slave node in real time, and distributes the crawling tasks to each Slave node in an orderly manner for parallel task processing. Each Slave node is responsible for parallel processing of tasks and completes the specific crawling tasks of the cloud platform web spider system. There is no network communication between the Slave nodes, and they all communicate with the Master node. The Master node is responsible for the task allocation and scheduling of the system as the central node. This centralized topological structure is simple to implement and convenient to manage.

[0115] The cloud platform web spider system of the present invention adopts the MapReduce programming mode based on the Hadoop framework. The Master node first divides the total crawling task into a series of subtasks and distributes them to each Slave node. The cloud platform parallel crawling method is adopted to improve the crawling efficiency. The Slave node completes each specific crawling subtask and then submits the crawled information to the Master node. The Master node aggregates the crawled information of each node and saves it in the file system HDFS based on the Hadoop framework. The overall framework model of the cloud platform web spider system proposed by the present invention is as follows: Figure 2 shown.

[0116] The overall framework model of the cloud platform web spider system of the present invention:

[0117] First, the JobTracker process is created and generated on the Master node. It divides the seed URL set and distributes the URL crawling task to each Slave node based on the running status of each Slave node. The JobTracker process is also responsible for real-time monitoring of the associated running status of the system. The JobTracker is deployed on a separate computer node in the cluster and does not participate in the specific crawling work.

[0118] Second, the JobTracker process distributes the segmented web page link URL to each TaskTracker process. Each TaskTracker process runs on each Slave node. After receiving the crawling subtask distributed to it by the JobTracker process, the TaskTracker process starts the corresponding Map task to perform the crawling of web page information. The corresponding Map task starts multiple threads to crawl web page information. After the Map task is completed, the crawled information is transmitted to the Reduce task as an intermediate result in the form of <link URL, content data> key-value pairs.

[0119] Third, after the Map task outputs the intermediate results, the TaskTracker process will start the Reduce task to complete the web page analysis, link deduplication filtering and merging operations, and store the link URL and web page content information obtained based on the web page content analysis in the link URL library and parsed web page library in the file system HDFS respectively. The TaskTracker process will also continue to send heartbeats to the JobTracker process through RPC to report on the resource usage and task running status on each node.

[0120] The basic process of the cloud platform web spider system is as follows Figure 3 As shown in the figure, what is given is the operation of a complete crawling process. During the parallel operation of the cloud platform, the system can completely run in parallel and asynchronously when performing web crawling, web analysis and link deduplication filtering modules. The present invention uses the computing model MapReduce based on the Hadoop framework to complete the cloud platform operation, and uses the file system HDFS based on the Hadoop framework to complete the storage of the web page information library and the link URL library, and accordingly designs and completes the cloud platform web spider system based on Hadoop.

[0121] The basic execution process of the cloud platform web spider system of the present invention is:

[0122] First, the user selects some high-quality initial URL seed sets based on the related subject information they want to obtain and puts them into the seed URL file. The initial URL seed set serves as the starting point for the system to add searches. The system selects URL links in the seed URL file to crawl web page information.

[0123] Second, after obtaining the corresponding URL link, the system establishes a network connection with the Web server corresponding to the URL link. If the network connection fails to be established and the waiting time exceeds a certain time, the system abandons the network connection and marks the URL link, and selects the next URL link from the URL link queue for access;

[0124] Third, if network communication with the Web server is successfully established, the system uses the MapReduce cloud platform computing model to crawl the information in the web page based on the http protocol, and stores the crawled information in the file system HDFS;

[0125] Fourth, after the webpage crawling is completed, the system further analyzes the webpage information and stores the parsed webpage content information in the parsed webpage library in the file system HDFS;

[0126] Fifth, parse the URL links contained in the web page and perform a deduplication and filtering operation on the links;

[0127] Sixth, store the URL links after link deduplication filtering in the link outbound URL library in the file system HDFS for future crawling work;

[0128] Seventh, if the conditions for the web spider to stop are not met, the system will evaluate the comprehensive value of each URL link based on the improved web spider model, and select a URL link with the highest priority, that is, the web page most relevant to the specified topic, for the next step of crawling.

[0129] 2. Improvement of Cloud Platform Web Spider

[0130] (I) Principles of Web Spiders

[0131] Web spiders crawl web page information related to a specified topic from a large amount of web page resources on the Internet, and filter out pages with low relevance to the topic. Currently, many professionals in many industries are troubled by the low accuracy of existing search engine results in the retrieval of topic network information. The demand for topic-based search engines is increasing day by day, and the core of topic-based search engines is web spiders. Web spiders focus more on topic-guided crawling. Based on the pre-specified topics, they crawl page information related to the topic based on web page analysis algorithms and topic relevance calculation models. Web spiders do not pursue wide coverage of web page information, but rather pursue the accuracy of web page information.

[0132] The web spider first analyzes the initial URL in the URL seed set, and filters the URL links with low relevance to the subject based on a specific web page analysis algorithm, and saves the URL links with high relevance to the subject and adds them to the list of URLs to be crawled. After crawling the web page content, the web spider selects a URL link with high relevance to the subject from the URL list according to a specific URL search method to crawl, and crawls the web page information in a cycle. Once the termination condition of the web spider is reached, the algorithm ends.

[0133] (II) Improved web spider model

[0134] The key technology of web spiders lies in web analysis algorithms and web spider crawling methods. The HITS algorithm and PageValue algorithm are the two most commonly used algorithms in search engine link analysis. The web analysis algorithms based on link structure represented by these two algorithms calculate the authority or weight value of web pages by analyzing the mutual links of web pages on the Internet, and then determine the priority of the web spider system to crawl the web URL link next. The PageValue algorithm is only suitable for discovering authoritative web pages but not for discovering web pages associated with the topic. The HITS algorithm will cause the searched page information to deviate from the associated topic. However, in the cloud platform web spider system proposed in the present invention, it is necessary to consider the information on the web page and the relevance of the link to the topic, so the use of these two algorithms alone does not meet the requirements of the system. On the other hand, the web analysis algorithm based on web page content only determines the priority of the search URL link based on the feature similarity of the web page content, and ignores the important role of link information in the topic relevance search process, and thus does not meet the topic relevance requirements of the cloud platform web spider system proposed in the present invention.

[0135] Through the above analysis, it can be seen that only using a web page analysis algorithm based on link structure or only using a web page analysis algorithm based on web page content cannot effectively make an accurate value assessment of the URL link. In order to meet the subject relevance requirements of the cloud platform web spider system proposed in the present invention, the present invention combines the HITS algorithm, a web page analysis algorithm based on link structure, and a web page analysis algorithm based on the subject similarity calculation of the VSM vector space model, and proposes an improved cloud platform web spider model. Based on the respective advantages and disadvantages of two different types of web page analysis algorithms, on the one hand, the relevance between web page content and the subject is evaluated from the aspect of web page content relevance, and on the other hand, the authority of the web page and the predictability of the related subject page are evaluated from the aspect of link analysis. The model comprehensively and objectively evaluates the comprehensive value of each URL link through the optimization algorithm, thereby improving the accuracy and efficiency of the cloud platform web spider system proposed in the present invention in crawling related subject resources.

[0136] Improved web spider model such as Figure 1 Its main design is:

[0137] First, the cloud platform web spider system is given a seed URL set, and then the URL segmentation module extracts the URL links in the seed URL set to perform URL segmentation operations, and then stores them in the un-crawled URL library in the cloud platform file system HDFS;

[0138] Second, the webpage crawling module reads the corresponding URL link from the uncrawled URL library and crawls it, and stores the crawled webpage information in the Web initial webpage library located in the cloud platform file system HDFS;

[0139] Third, the web page parsing module parses the URL links contained in the web page and stores them in the link URL library in the cloud platform file system HDFS, and stores the parsed web page content information in the parsed web page library in the cloud platform file system HDFS;

[0140] Fourth, the hyperlink evaluator reads the outbound URL library in the cloud platform file system HDFS, and calculates the Hub weight and Authority weight of each URL link based on the HITS algorithm. At the same time, it constructs a topic description matrix and obtains the vector representation of the topic from the topic description matrix and the topic web page set. It combines word frequency statistics and content structure information to represent the web page vector, and uses the cosine angle theorem to obtain the relevance value for the topic vector and the web page vector. The page relevance evaluator performs word segmentation processing on the parsed web page content information based on the associated topic feature word library and counts the feature topic word frequency, and then calculates the topic relevance of the web page content based on the VSM vector space model.

[0141] Fifth, the web spider comprehensive value evaluator adopts the improved web spider model algorithm to calculate the comprehensive value of each URL link based on the link value and content theme relevance value, compares the web links in the crawling queue, and determines the order in which the web spider will crawl the URLs next.

[0142] (III) Improved web spider model algorithm implementation

[0143] The core of a web spider is to make the page information corresponding to the crawled URL link have a high degree of subject relevance. The present invention proposes an improved web spider model and analyzes in detail how to evaluate the comprehensive value of the URL link. The improved web spider model is implemented as follows.

[0144] 1. Calculate the value of hyperlink structure

[0145] The improved web spider model uses the hyperlink analysis algorithm HITS to calculate the hyperlink structure value of each candidate URL, and calculates the Hub weight and Authority weight for each URL link. These two values ​​will be used in the calculation of the comprehensive value of the web page. i , the calculation of its Hub weight value and Authority weight value is shown in Formula 1 and Formula 2:

[0146]

[0147]

[0148] Among them, v i , v j∈B represents v i Exists to v j Hyperlink, v j , v i ∈C represents v j Exists to v i A[v i ]、H[v i ] indicates v i The Hub weight and Authority weight are calculated through continuous iteration. In the first step of the iterative calculation, each URL link is assigned an initial value:

[0149] A[V i ]=1,H[V i ]=1 Formula 3

[0150] Finally, after iterative calculation, A[v i ]、H[v i ] Normalized, the calculation formula is formula 4 and formula 5:

[0151]

[0152]

[0153] Among them, v i , v j ∈B represents v j Exists to v i Hyperlink, v j , v i ∈C represents v j Exists to v i Hyperlink.

[0154] 2. Calculate the topic relevance of the content

[0155] The improved web spider model of the present invention uses the TF-IDF formula based on the VSM vector space model to calculate the topic relevance of web page information content. According to the vector space model description, for a web page q, it is formally represented as a feature word weighted vector Q, and the calculation formula is Formula 6:

[0156] Q={q1,q2,…,q i ,…,q m ) Formula 6

[0157] Among them, q1 represents the feature word d i The weighted value in page Q is calculated by the TF-IDF formula. The weighted value is calculated as Formula 7, where n is the dimension of the feature word vector space of the topic:

[0158] qi =tf i *idf i Formula 7

[0159] Among them, tf i Indicates the characteristic word d i The frequency of the term in document Q, idf i Indicates the characteristic word d i The inverse document frequency of is calculated as formula 8:

[0160]

[0161] Among them, m i Indicates the characteristic words d that appear in the sample page set i The topic relevance of page q is calculated by calculating the weighted vector Q of the feature word of the page and the topic feature vector U + The inner product of two vectors gives the result, calculated as Equation 9:

[0162]

[0163] Where m represents the eigenvector U + The dimension of , sim(q) represents the topic relevance of page q. The larger its value is, the greater the probability that the web page information of page q is associated with the topic. The improved web spider model proposed by the present invention does not directly define whether page q is associated or not, but calculates the probability of the web page information being associated with the topic, and then calculates the comprehensive value of the URL link based on this probability value, so that the evaluation of the URL link is more comprehensive and accurate.

[0164] 3. Evaluate the comprehensive value of the web page

[0165] The present invention improves the Shark-Search algorithm on the basis of using the vector space model to calculate the topic relevance of web page information content, and adds the relevance information of the link structure therein, and the calculation formula is Formula 10:

[0166]

[0167] Where sim(i) represents the topic relevance of web page i, fa(i) represents the set of all web pages linking to web page i, Inde(i) represents the number of backlinks of web page i, that is, the number of web pages in fa(i), the value range of variable q is between 0 and 1, and variable q adjusts the proportion of link structure and web page text, te_sim(i) is the similarity between web page content and topic, and the specific calculation is shown in formula 11:

[0168] te_sim(i)=Sim(Q_Text(i),Th)×p

[0169] +Sim(Q_Tit(i),Th)×(1-p) Formula 11

[0170] Q_Text(i) is the main text content of web page i, Q_Tit(i) is the title information of web page i, Th represents the keyword set of the topic, and the variable p is the same as q, and its value range is between 0 and 1. The calculation of Sim uses the TF-IDF algorithm based on the vector space model to calculate the feature vector of the document, and then uses the cosine theorem to calculate the similarity, as shown in Formula 9.

[0171] Based on the calculation of the value of the hyperlink structure and the calculation of the subject relevance of the content, the improved web spider model algorithm of the present invention comprehensively considers the link value of web page q and the subject relevance of the content, and provides a calculation formula for calculating the comprehensive value of the web page, which is Formula 12:

[0172] Value(i)=(H[i]+A[i])×(s+sim(i)) Formula 12

[0173] Among them, Value(i) is the comprehensive value of the web page q to be finally obtained, H[i] and A[i] represent the Hub weight value and Authority weight value of web page i respectively, which are the results calculated by using Formula 4 and Formula 5 based on the HITS algorithm, and sim(i) represents the topic relevance of web page i, which is the result calculated by using Formula 10. In addition, a control factor s is added to Formula 12, and its value range is between 0 and 1. This design is because there is a special case that some pages with high Hub weight values ​​and Authority weight values ​​may point to another set of topic pages, but their own web page content is not related to the topic. At this time, it is necessary to add a control factor to enable the web spider to crawl more topic-related pages.

[0174] The present invention proposes an improved web spider model algorithm. The algorithm model is improved by the HITS algorithm, a web page analysis algorithm based on link structure, and the web page analysis algorithm based on the VSM vector space model for topic similarity calculation. On the one hand, the relevance between web page content and the topic is evaluated from the aspect of web page content relevance. On the other hand, the authority of the web page and the predictability of the related topic page are evaluated from the aspect of link analysis, so as to evaluate the comprehensive value of the web page information.

[0175] 3. Storage structure of cloud platform web spider on HDFS

[0176] The present invention summarizes the storage structure adopted by the cloud platform web spider system, designs the corresponding data structure according to the characteristics of the file system HDFS, and proposes the uncaptured link library, initial web page library, parsed link library, and parsed web page library adopted by the system based on the overall framework model of the system.

[0177] (I) Link library not captured

[0178] The uncrawled URL library stores the currently pre-crawled URLs, which are stored in the file system HDFS after the URL segmentation module processes the URLs read from the URL seed file. The web crawling module obtains URLs from the uncrawled URL library to perform specific web crawling work.

[0179] (II) Initial Web Page Library

[0180] The initial web page library stores the initial web page information captured by each node in the cloud platform web spider by the web crawling module. After processing, they are saved in the file system HDFS for use by subsequent web page parsing modules.

[0181] (III) Parsing link library

[0182] The parsing link library stores the link parsing data obtained by parsing the initial web page information captured by each node in the cloud platform web spider through the web parsing module. After processing, they are saved in the file system HDFS for subsequent crawling work.

[0183] (IV) Parsing Web Page Library

[0184] The parsed web page library stores the page parsing data obtained by parsing the initial web page information captured by each node in the cloud platform web spider through the web page parsing module. After processing, they are saved in the cloud platform file system HDFS for subsequent use.

[0185] The above four storage structures are implemented on the Hadoop-based file system HDFS for the function modules in the cloud platform web spider system to call.

[0186] 4. Implementation of MapReduce Algorithm for Web Spider on Cloud Platform

[0187] (I) System cloud platform architecture

[0188] The system is divided into three parts: spider nodes, master nodes, and HDFS. The master node coordinates the crawling between spider nodes. Spider nodes crawl pages and store the crawled pages in HDFS. The cloud platform architecture of the system is as follows: Figure 4 shown.

[0189] The working mechanism of information and data interaction between the master node and the spider sub-node is:

[0190] First, the master control node knows how many spiders are deployed in the entire system based on its own configuration information, and starts the spiders by remote login from the master control node;

[0191] Second, after the spider child node is started, it sends a "Ready" status message to the master control node, indicating that the spider is ready to accept crawling tasks;

[0192] Third, after receiving the "Ready" message from the spider child node, the master control node generates an initial task, that is, encapsulating some URLs of the web pages to be crawled, and sending them to a specific path of the spider child node in the form of a file. Then, it sends a "Working" reply message to the spider node. If the master control node finds that there is no data in the current database, that is, there is no task to send, and there are still spiders crawling in the system, it will send a "Wait" reply message to the spider. If the spider system is started for the first time or all spiders are in the "Wait" state, it means that the user has not added a crawling task or the crawling is completed. The master control node will send a "Stop" message to each spider.

[0193] Fourth, after receiving the reply message of "Working", the spider child node goes to the corresponding path and loads the seed URL into the crawling queue. If it is the first start, it starts the crawling thread to start the crawling task. If it is not the first start, it continues to crawl the next task after the current crawling is completed. If the spider receives the "Wait" message, it sets the state to "Wait". After crawling the current task, it is in a waiting state until the master node wakes it up. If it receives the "Stop" message, the spider exits directly.

[0194] Fifth, in the process of crawling pages, when the extracted URLs reach a certain amount, they will be encapsulated into a data file, that is, written into a text file, and the spider will send it to a specific path of the master node and send the status information of "Dada" to the master node;

[0195] Sixth, after receiving the "Dada" message, the master control node loads the data into the database under the specified path;

[0196] Seventh, when the spider is about to complete the crawling task, it will also send a "Ready" status message to the main control node, and then return to the first interactive state.

[0197] (II) Improved task allocation algorithm

[0198] In the cloud platform spider system, the balanced distribution of crawling tasks is a key issue that affects system performance and resource allocation. The existing technology mostly adopts a centralized task segmentation method. This centralized task segmentation method often leads to repeated collection of web pages, resulting in a waste of time and bandwidth. The biggest disadvantage of the centralized task segmentation method is that the coordinator has to forward a large number of URLs, which becomes a bottleneck for system performance. To solve this problem, a secondary hash mapping algorithm is used to support cloud platform URL allocation, and the system scale is dynamically scalable, so that the system can perform efficient and balanced cloud platform URL allocation, and the system has good dynamic configurability.

[0199] Assume that within a crawling cycle, the maximum number of logical nodes is logic_num. The number of currently running physical nodes is physic_num. Each crawling node has two tables: one is a physical node table, which is used to store the information of each node in the current system; the other is a logical node table, which stores the information of logic_num logical nodes. If the node corresponding to the node does not exist, the value is 0. After the URL is standardized, the URL is first hash-mapped to the element in the corresponding logical table. If the element is not 0, the ID number is taken out to determine whether to route the URL to other crawling nodes. Otherwise, all URLs on the element are evenly distributed to the current physical nodes of the system through a second hash mapping. At this time, the ID number of the corresponding crawling node is obtained and it is determined whether to route the URL. In addition, in order to distribute the URLs to each crawling node as evenly as possible, a hash function with good performance needs to be selected. To this end, the present invention designs a URL hash function:

[0200]

[0201] In this function, in the first hash mapping, Q=logic_num, i.e., the number of logical nodes, and in the second hash mapping, Q=physic_num, i.e., the number of physical nodes. The Ascii function takes the ASCII code of each character in the string.

[0202] However, the task partitioning method based on secondary hash mapping only solves the problem of uniform distribution, and does not take into account the load conditions of each spider sub-node. Therefore, the present invention proposes an improved task allocation algorithm that takes into account both uniform distribution and the load conditions of each spider sub-node.

[0203] After the secondary hash mapping algorithm calculates the crawling sub-nodes, the improved task splitting algorithm uses the weighted minimum connection scheduling algorithm to determine whether the node's load allows the allocation of new URL crawling tasks. Each sub-node uses a corresponding weight value to represent its processing performance. The default weight value is set to 1. The system administrator dynamically sets the server's weight value. When scheduling new connections, the weighted minimum connection scheduling tries to make the number of established connections on the server proportional to its weight value.

[0204] The algorithm flow of weighted minimum connection scheduling is as follows: Assume that there is a group of servers K = {K0, K1, ..., K n-1 ), J(K i ) indicates server K i The weight value, F(K i ) indicates server K i The current number of connections of all servers is F. sum =∑F(K i )(i=0,1,…,nl), the current new connection request will be sent to server K m , if and only if server K m The URL seed is sent only when the following conditions are met:

[0205]

[0206] Among them, J(K i ) is not 0, the log of the child node is fed back to the main control node at a certain interval, and the number of child server connections F(K i ) is obtained by reading the log, comparing the ratio of the number of child node connections and the prior weight value, obtaining the child node with the smallest load, and assigning a new crawling task.

[0207] (III) Algorithm implementation of functional modules

[0208] The module design of the cloud platform web spider system is as follows Figure 5 The modules of the cloud platform web spider system proposed in the present invention are divided into:

[0209] Module 1, URL segmentation module: Users select some high-quality initial URL seed sets based on the related topic information they want to obtain and put them into the seed URL file. The initial URL seed set serves as the starting point for the cloud platform web spider system to add searches, and plays a vital role in the system's crawling of web pages with related topics. After obtaining the URL list, the URL segmentation module is used to segment the seed URL set and assign it to the TaskTracker process on each Slave node for crawling.

[0210] Module 2, web crawling module: based on the URL link obtained from the uncrawled URL library in the cloud platform file system HDFS and crawling web page information. First, make an HTTP connection with the web server where the URL is located, and then download the web page information and store it in the initial web page library in the cloud platform file system HDFS, waiting for the web page parsing module to process it.

[0211] Module three, web page parsing module: After the web crawling module completes the crawling work, it saves the web page information to the initial web page library of the cloud platform file system. Then the web page parsing module will distribute the tasks according to the number of blocks of the input data, that is, the number of blocks of the page content in the initial web page library of the file system HDFS. Then the web page parsing module uses the MapReduce cloud platform computing model to parse the web page content, including web page content information and link URL information. Finally, the web page parsing module parses the URL links contained in the network pages and stores them in the link URL library in the file system HDFS, and stores the parsed web page content information in the parsed web page library in the file system HDFS.

[0212] Module 4, link filtering module: The link URLs obtained by the web page parsing module are filtered through a link deduplication operation. For URL links that do not meet the standards, they will be normalized before they can be used for subsequent crawling work. For duplicate URL links, deduplication operations are required.

[0213] Module five, data storage module: The data storage module stores the parsed web page information and the URL links after filtering and deduplication in the file system HDFS for subsequent crawling work. After the cloud platform web spider system is started, it performs the tasks of URL segmentation, web page crawling, web page content parsing and link deduplication filtering function modules, and continues to run in a loop in parallel with the cloud platform until the cloud platform web spider system reaches the corresponding crawling end conditions.

[0214] 1. URL segmentation module

[0215] The URL segmentation module segments the URLs in the seed URL set, which is the first step for the cloud platform web spider system to officially start; first, the URL segmentation module obtains the seed URL from the seed URL set; then, after completing the segmentation task, it distributes it to each Slave node for the crawling work of the web crawling module. The spiders on each Slave node store the distributed URL set in their respective uncrawled URL library, and then select a URL web link from the uncrawled URL library as the start of a specific crawling task. The most important task of the system is to ensure that the crawling tasks on each node do not conflict. At this time, task allocation is particularly important. Therefore, after the URL segmentation module segments the seed URL set, the spiders on each TaskTracker each obtain the URL set assigned to it. The spiders on each TaskTracker cooperate with each other and perform crawling tasks in parallel to avoid conflicts between the crawling tasks on each TaskTracker. The job of the URL segmentation module is to segment the URL list into several fragments and save them in the uncrawled URL library in the file system HDFS for the web crawling module to use.

[0216] The MapReduce model algorithm of the URL segmentation module is described as follows:

[0217] The first step is initialization: create a new MapReduce task and perform initialization operations on it;

[0218] The second step is preprocessing: preprocess the data through InputFormat and transform it into<key,value> The key-value pairs in this data format are convenient for MapReduce tasks to process;

[0219] The third step is data sharding: pre-processing the seed URL set, that is, obtaining URL link information from the seed URL set and sharding it;

[0220] Step 4: Map process: perform associated operations on each URL according to the improved task allocation algorithm;

[0221] Step 5, Combiner process: merge the values ​​of the same keyword key in the intermediate file output in step 4 to improve the processing efficiency of subsequent operation steps.

[0222] Step 6, Partitioner process: After the merging process in step 5, the intermediate results of the output of the Map task are split into several regions based on the key using a hash table.

[0223] Step 7, Reduce process: Based on the calculation results of the Map task, each Reduce task outputs the task allocation results of the URL links in the seed URL set;

[0224] Step 8. Store in HDFS: Process the output results of Reduce through OutputFormat, and store the URLs assigned to each crawling sub-node in the uncrawled link library in its file system HDFS for the next crawling work.

[0225] 2. Web crawling module

[0226] The web crawling module obtains URL links from the uncrawled URL library to perform formal web crawling. The web crawling module is a key core module. The web crawling module can perform parallel operations on crawling tasks. After entering the web crawling module, the TaskTracker working on each Slave node first starts multiple crawling threads to work together to crawl the web page information corresponding to the URL link based on the uncrawled URL set assigned to it by the URL segmentation module. After being processed by the MapReduce cloud platform computing model, the data obtained by the web crawling module is encapsulated and mapped into the "<web link, web data>" type<key,value> The original web page library is saved in the file system HDFS in the form of key-value pairs.

[0227] The MapReduce model algorithm of the web crawling module is described as follows:

[0228] Step 1, Initialization: Create a new MapReduce task and perform initialization operations on it;

[0229] Step 2, preprocessing: Use InputFormat to preprocess the data and transform it into<key,value> This data format is key-value pairs, which is convenient for MapReduce tasks to process;

[0230] Step 3, data segmentation: pre-process the output of the URL segmentation module, obtain URL link information from the uncaptured URL library, and segment it.<key,value> A key-value pair is a record. In this module, the key keyword represents the character offset of the URL string, and the value is the string information of the URL.

[0231] Step 4, Map process: This process is to convert the<key,value> The keyword key and value value of the key-value pair are exchanged, and then the intermediate result is output. The keyword key represents the string information of the URL, and the value value is the character offset of the URL string.

[0232] Step 5, Combiner process: merge the values ​​of the same keyword key in the intermediate files output in step 4 to improve the processing efficiency of subsequent operation steps;

[0233] Step 6, Partitioner process: After the merging process in step 5, the intermediate results of the output of the Map task are split into several zones based on the key using a hash table. Each zone in the web crawling module corresponds to the same server node.

[0234] Step 7, Reduce process: Based on the partitions in step 6, each Reduce task starts multiple download threads to crawl URLs in the same partition and outputs the crawled results;

[0235] Step 8. Store in HDFS: Process the output of Reduce through OutputFormat and save it in the initial web page library in the file system HDFS.

[0236] Among them, the specific web page crawling is assisted by the tool software HttpClient. First, the crawling thread uses the HttpClient class to establish network communication with the web server, and applies the HTTP post or get function to make a connection request. Then, the web server responds to the request and returns the web page information in the form of a response message. Then, the web page message header information is checked, including the web page content length, character encoding set, content type, and redirection address, and the response message information is obtained through the getMethod.getResponseBody() method; finally, the corresponding web page content information is obtained after parsing the response message, and the content of the Robots file is read during the web crawling process. Only those web page information that has permission or is allowed to be crawled are crawled, complying with the Robots protocol, and in the process of establishing a connection with the web server, the parameter TIMEOUT is set. The TIMEOUT parameter is the maximum time allowed in the process of establishing a connection. If a link fails to establish a connection within the time exceeding TIMEOUT, the link will be discarded.

[0237] 3. Web page parsing module

[0238] The web page parsing module includes two aspects: one is to parse the captured web page to obtain the content information of the web page, and the other is to parse the hyperlink URL information contained in the web page. Before parsing the web page, the type of web page is identified according to the characteristics of different types of web pages, and the corresponding method is selected for different types of web pages to parse different types of web pages. The steps of the web page parsing module are as follows: Figure 6 As shown. The parsing process for different types of web pages is as follows:

[0239] (1) Parsing HTML type web pages

[0240] First, the web page body content is extracted from the initial HTML file as the basis for subsequent processing. For HTML type web pages, the primary task of the web page parsing module is to analyze the structure of the web page.

[0241] The web page parsing module adopts the MapReduce cloud platform computing model. The Map() operation analyzes and filters various types of tags used for structured display of HTML type web pages, extracts the attributes and content of each tag, and stores the parsed content in the parsed web page library in the file system HDFS. Then, the Reduce() operation merges and summarizes all the link URLs extracted by the Map() operation and saves them in the link URL library in the file system HDFS for the spider to carry out subsequent crawling work.

[0242] The description of the MapReduce model algorithm of the web page parsing module is as follows:

[0243] Step 1: Initialization: Create a new MapReduce task and perform initialization operations on it;

[0244] Step 2: Preprocessing: Use InputFormat to preprocess the data and transform it into<key,value> This data format is key-value pairs, which is convenient for MapReduce tasks to process;

[0245] Step 3, data sharding: pre-process the data of the initial webpage library, that is, obtain the initial information of the URL page from the initial webpage library, and perform sharding on it, and process the split data into<key,value> A key-value pair is a record. In this module, the key keyword represents the initial information of the page corresponding to the URL characters, and the value is the string information of the URL.

[0246] Step 4, Map process: This process is to convert the<key,value> The keyword key and value value of the key-value pair are exchanged, and then the intermediate result is output. The keyword key represents the string information of the URL, and the value value is the initial information of the page corresponding to the URL characters;

[0247] Step 5: Combiner process, that is, merging the values ​​of the same keyword key in the intermediate file output in step 4 to improve the processing efficiency of subsequent operation steps;

[0248] Step 6, Partitioner process, after the merging process in step 5, splits the intermediate results of the Map task output into several zones based on the keyword key using a hash table.

[0249] Step 7, Reduce process: Based on the partitioning in step 6, each Reduce task will parse the initial page information, parse the content information of the web page and the URL information of the link, calculate the comprehensive value of each URL link based on the improved web spider model algorithm, and output the crawled results;

[0250] Step 8: Store in HDFS: Process the output results of Reduce through OutputFormat and save them in the parsing web page library and parsing link library in the file system HDFS.

[0251] Among them, the analysis of the HTML web page layout structure is assisted by the tool software HtmlParser. First, HtmlParser initializes all the content data in the web page, and then parses each tag separately, extracting the attributes and content of the Title, Body, and Link tags respectively. The parsed content will be stored in the parsed web page library in the file system HDFS, and the extracted web page link URL will be filtered and deduplicated by the link deduplication filtering module and then saved in the link outbound URL library in the file system HDFS for subsequent crawling work.

[0252] (2) Parsing non-HTML web pages

[0253] Different types of parser plug-ins are used to extract non-HTML web page data to complete data processing. The parsing methods for PDF and Office type web pages are as follows:

[0254] 1) PDFBox is used to parse PDF web pages. 2) POI is used to extract data from Microsoft Word, Excel, and PowerPoint documents. POI provides users with a set of Java APIs to extract data from these three types of web pages.

[0255] 4. Link deduplication filtering module

[0256] The present invention designs a link deduplication filtering module to realize the normalization and deduplication processing of URL links. The link deduplication filtering module includes two filter classes, namely: a URL normalization filter and a URL deduplication filter.

[0257] (1) URL normalization filter: The URL normalization filter mainly includes two aspects of work. On the one hand, it converts the URL link string obtained by the web page parsing module into lowercase characters; on the other hand, it deletes special characters contained in the URL key string obtained by the web page parsing module. When surfing the Internet, sometimes some URL link strings contain some special characters.

[0258] (2) URL deduplication filter: BloomFilter is used to eliminate duplicate URL links. The BloomFilter algorithm uses a very long binary bit vector and a series of random hash functions. The algorithm flow is as follows:

[0259] Step 1: Assume that the data set to be deduplicated is V{v1, v2, v3, …, v n ), and initialize a bit vector U of length m, with each bit set to zero;

[0260] Step 2: Construct k random hash functions with uniform address distribution: hash1(v), hash2(v), hash3(v), ..., hash k (v), the input of the function is the data v to be deduplicated, and the output result range is between 1 and m;

[0261] Step 3, insert operation: take a data in the set V as input, use these k hash functions to perform hash operations, obtain k hash addresses, and assign the contents of the corresponding k addresses in the bit vector U to 1;

[0262] Step 4, query operation: k hash functions perform hash operations on the data to be queried, obtain k addresses, and detect the values ​​of the corresponding k addresses in the vector U. If all are 1, the element exists, otherwise the element does not exist.

[0263] The present invention proposes an overall framework model and basic process of a cloud platform web spider system, designs a corresponding data structure according to the characteristics of the cloud platform file system HDFS, and uses the cloud platform computing model MapReduce of the Hadoop framework and the cloud platform file system HDFS to implement the algorithms of various functional modules. In the process of implementing the cloud platform web spider on the cloud platform, an improved task allocation algorithm is proposed, which can take into account the uniform distribution and the load of each crawling sub-node, improve the system performance, optimize the resource allocation, and greatly improve the crawling efficiency and accuracy of the cloud platform web spider system.

Claims

1. A web spider theme search system based on an improved cloud platform, characterized in that: include: First, the HITS algorithm based on link structure and the VSM vector space model-based topic similarity calculation algorithm are improved. On the one hand, the relevance between web page content and topic is evaluated from the aspect of web page content relevance, and on the other hand, the authority of web pages and the predictability of related topic pages are evaluated from the aspect of link analysis. An improved web spider model algorithm is proposed to evaluate the comprehensive value of web page information. Second, in the process of cloud platform implementation of cloud platform web spider, an improved task allocation algorithm is proposed. The secondary hash mapping algorithm is used to support cloud platform URL allocation, and the system scale is dynamically scalable, so that the system can perform efficient and balanced cloud platform URL allocation, and the system has dynamic configurability, taking into account the uniform distribution and the load of each crawling sub-node, so as to improve the crawling rate and accuracy of the cloud platform web spider system. Third, a general framework model of cloud platform web spider based on Hadoop is proposed, and the storage structure of cloud platform web spider is designed and implemented on the file system HDFS, and the MapReduce algorithm is implemented for each functional module based on module division. Fourth, the cloud platform web spider system based on Hadoop is implemented and tested. The execution process of the cloud platform web spider system is: First, the user selects a high-quality initial URL seed set based on the related subject information he wants to obtain and puts it into the seed URL file. The initial URL seed set serves as the starting point for the system to add search, and the system selects URL links in the seed URL file to crawl web page information; Second, after obtaining the corresponding URL link, the system establishes a network connection with the Web server corresponding to the URL link. If the network connection fails to be established and the waiting time exceeds the set time, the system abandons the network connection and marks the URL link, and selects the next URL link from the URL link queue for access; Third, if network communication with the Web server is successfully established, the system uses the MapReduce cloud platform computing model to crawl the information in the web page based on the http protocol, and stores the crawled information in the file system HDFS; Fourth, after the webpage crawling is completed, the system further analyzes the webpage information, evaluates the relevance of the webpage content to the subject from the aspect of webpage content relevance, evaluates the authority of the webpage and the predictability of the related subject page from the aspect of link analysis, evaluates the value of the webpage information, and stores the parsed webpage content information in the parsed webpage library in the file system HDFS; Fifth, parse the URL links contained in the web page and perform a deduplication and filtering operation on the links; Sixth, store the URL links after link deduplication filtering in the link outbound URL library in the file system HDFS for future crawling work; Seventh, if the conditions for the web spider to stop are not met, the system will evaluate the comprehensive value of each URL link according to the improved web spider model, and select a URL link with the highest priority, that is, the web page most relevant to the specified topic, for the next crawling work; The improved web spider model is designed as follows: First, the cloud platform web spider system is given a seed URL set, and then the URL segmentation module extracts the URL links in the seed URL set to perform URL segmentation operations, and then stores them in the un-crawled URL library in the cloud platform file system HDFS; Second, the webpage crawling module reads the corresponding URL link from the uncrawled URL library and crawls it, and stores the crawled webpage information in the Web initial webpage library located in the cloud platform file system HDFS; Third, the web page parsing module parses the URL links contained in the web page and stores them in the link URL library in the cloud platform file system HDFS, and stores the parsed web page content information in the parsed web page library in the cloud platform file system HDFS; Fourth, the hyperlink evaluator reads the outbound URL library in the cloud platform file system HDFS, and calculates the Hub weight and Authority weight of each URL link based on the HITS algorithm. At the same time, it constructs a topic description matrix and obtains the vector representation of the topic from the topic description matrix and the topic web page set. It combines word frequency statistics and content structure information to represent the web page vector, and uses the cosine angle theorem to obtain the relevance value for the topic vector and the web page vector. The page relevance evaluator performs word segmentation processing on the parsed web page content information based on the associated topic feature word library and counts the feature topic word frequency, and then calculates the topic relevance of the web page content based on the VSM vector space model. Fifth, the web spider comprehensive value evaluator uses the improved web spider model algorithm to calculate the comprehensive value of each URL link based on the link value and content theme relevance value, compares the web links in the crawling queue, and determines the order of the web spider to crawl the URLs next; The overall framework model of the cloud platform web spider: First, the JobTracker process is created and generated on the Master node. It divides the seed URL set and distributes the URL crawling task to each Slave node based on the running status of each Slave node. The JobTracker process is also responsible for real-time monitoring of the associated running status of the system. The JobTracker is deployed on a separate computer node in the cluster and does not participate in the specific crawling work. Second, the JobTracker process distributes the segmented web page link URL to each TaskTracker process. Each TaskTracker process runs on each Slave node. After receiving the crawling subtask distributed to it by the JobTracker process, the TaskTracker process starts the corresponding Map task to perform the crawling of web page information. The corresponding Map task starts multiple threads to crawl web page information. After the Map task is completed, the crawled information is transmitted to the Reduce task as an intermediate result in the form of <link URL, content data> key-value pairs. Third, after the Map task outputs the intermediate results, the TaskTracker process will start the Reduce task to complete the web page analysis, link deduplication filtering and merging operations, and store the link URL and web page content information obtained based on the web page content analysis in the link URL library and parsed web page library in the file system HDFS respectively. The TaskTracker process will also continue to send heartbeats to the JobTracker process through RPC to report the resource usage and task running status on each node; The improved web spider model uses the hyperlink analysis algorithm HITS to calculate the hyperlink structure value of each candidate URL, and calculates the Hub weight and Authority weight for each URL link. These two values ​​will be used in the calculation of the comprehensive value of the web page. Based on the HITS algorithm, the URL link v i , the calculation of its Hub weight value and Authority weight value is shown in Formula 1 and Formula 2: Among them, v i , v j ∈B represents v i Exists to v j Hyperlink, v j , v i ∈C represents v j Exists to v i Hyperlink, A[v i ]、H[v i ] indicates v i The Hub weight value and Authority weight value are calculated through continuous iteration. In the first step of the iterative calculation, each URL link is assigned an initial value: A[V i = -1, H[V i = 1 Equation 3 Finally, after iterative calculation, A[v i ]、H[v i ] Normalized, the calculation formula is formula 4 and formula 5: Among them, v i , v j ∈B represents v j Exists to v i Hyperlink, v j , v i ∈C represents v j Exists to v i Hyperlinks; The improved web spider model uses the TF-IDF formula based on the VSM vector space model to calculate the topic relevance of web page information content. According to the vector space model description, for a web page q, it is formally represented as a feature word weighted vector Q, and the calculation formula is Formula 6: Q = {q1, q2, …, q i , …, q m ) Equation 6 Among them, q1 represents the feature word d i The weighted value in page Q is calculated by the TF-IDF formula. The weighted value is calculated as Formula 7, where n is the dimension of the feature word vector space of the topic: q i = tf i *idf i formula 7 Among them, tf i Indicates the characteristic word d i The frequency of the term in document Q, idf i Indicates the characteristic word d i The inverse document frequency of is calculated as formula 8: Among them, m i Indicates the characteristic words d that appear in the sample page set i M represents the total number of pages in the sample page set. The topic relevance of page q is calculated by calculating the weighted vector Q of the feature word of the page and the topic feature vector U + The inner product of two vectors gives the result, calculated as Equation 9: Where m represents the eigenvector U + , sim(q) represents the topic relevance of page q. The larger its value is, the greater the probability that the web page information of page q is associated with the topic. The improved web spider model does not directly define whether page q is associated or not, but calculates the probability of the web page information being associated with the topic, and then calculates the comprehensive value of the URL link based on this probability value. Improved task allocation algorithm: In a crawling cycle, the maximum number of logical nodes is logic_num, and the number of currently running physical nodes is physi_num. Each crawling node has two tables: one is the physical node table, which is used to store the information of each node in the current system; the other is the logical node table, which stores the information of logic_num logical nodes. If the node corresponding to the node does not exist, the value is 0. After the URL is standardized, the URL is first hashed to the element in the corresponding logical table. If the element is not 0, the ID number is taken out to determine whether to route the URL to other crawling nodes. Otherwise, all URLs on the element are evenly distributed to the current physical nodes of the system through the second hash mapping. At this time, the ID number of the corresponding crawling node is obtained and it is determined whether to route the URL. In addition, to evenly distribute the URLs to each crawling node, a hash function with good performance needs to be selected. For this purpose, a URL hash function is designed: In this function, in the first hash mapping, Q = logic_num, i.e., the number of logical nodes, and in the second hash mapping, Q = physic_num, i.e., the number of physical nodes. The Ascii function takes the ASCII code of each character in the string; An improved task allocation algorithm that takes into account both uniform distribution and the load of each spider node; After the secondary hash mapping algorithm calculates the crawling sub-nodes to be assigned, the improved task splitting algorithm uses the weighted minimum connection scheduling algorithm to determine whether the load of the node allows the new URL crawling task to be assigned. Each sub-node uses a corresponding weight value to represent its processing performance. The default weight value is set to 1. The system administrator dynamically sets the weight value of the server. The weighted minimum connection scheduling makes the number of established connections of the server proportional to its weight value when scheduling new connections. The algorithm flow of weighted minimum connection scheduling is as follows: there is a group of servers K = {K0, K1, ..., K n-1 ), J(K i ) indicates server K i The weight value, F(K i ) indicates server K i The current number of connections of all servers is F. sum =∑F(K i )(i=0,1,…,nl), the current new connection request will be sent to server K m , if and only if server K m The URL seed is sent only when the following conditions are met: Among them, J(K i ) is not 0, the log of the child node is fed back to the main control node at regular intervals, and the number of child server connections F(K i ) Obtained by reading the log, comparing the ratio of the number of sub-node connections and the prior weight value, obtaining the sub-node with the minimum load, and assigning a new crawling task; The modules of the cloud platform web spider system are divided into: Module 1, URL segmentation module: Users select some high-quality initial URL seed sets based on the related topic information they want to obtain and put them into the seed URL file. The initial URL seed set is used as the starting point for the cloud platform web spider system to add searches, which is crucial for the system to crawl web pages of related topics. After obtaining the URL list, the URL segmentation module segments the seed URL set and assigns it to the TaskTracker process on each Slave node for crawling; Module 2, web crawling module: based on the URL link obtained from the un-crawled URL library in the cloud platform file system HDFS and crawling web page information, first establish an HTTP connection with the web server where the URL is located, and then download the web page information and store it in the initial web page library in the cloud platform file system HDFS, waiting for the web page parsing module to process; Module three, web page parsing module: after completing the crawling work, the web page crawling module saves the web page information to the initial web page library of the cloud platform file system. Then the web page parsing module distributes the tasks according to the number of blocks of input data, that is, the number of blocks of page content in the initial web page library of the file system HDFS. Then the web page parsing module uses the MapReduce cloud platform computing model to parse the web page content, including web page content information and link URL information. Finally, the web page parsing module parses the URL links contained in the web page and stores them in the link URL library in the file system HDFS, and stores the parsed web page content information in the parsed web page library in the file system HDFS. Module 4, link filtering module: The link URLs obtained by the web page parsing module are filtered to remove duplicate links. For URL links that do not meet the standards, they will be normalized before they can be used for subsequent crawling work. For duplicate URL links, deduplication operations are required. Module 5, Data Storage Module: The data storage module stores the parsed web page information and the URL links after filtering and deduplication in the file system HDFS for subsequent crawling work. After the cloud platform web spider system is started, it performs the tasks of URL segmentation, web page crawling, web page content parsing and link deduplication filtering function modules, and continues to run in a loop in parallel with the cloud platform until the cloud platform web spider system reaches the corresponding crawling end conditions; Web crawling module: First, TaskTracker on each Slave node starts multiple crawling threads to work together to crawl the web page information corresponding to the URL link based on the uncrawled URL set assigned to it by the URL segmentation module. After being processed by the MapReduce cloud platform computing model, the data obtained by the web crawling module is encapsulated and mapped into the "<web page link, web page data>" type.<key,value> The original web page library is saved in the file system HDFS in the form of key-value pairs; The MapReduce model algorithm of the web crawling module is described as follows: Step 1, Initialization: Create a new MapReduce task and perform initialization operations on it; Step 2, preprocessing: Preprocess the data through InputFormat and transform it into<key,value> This data format is key-value pairs, which is convenient for MapReduce tasks to process; Step 3, data sharding: pre-process the output of the URL sharding module, obtain the URL link information from the uncaptured URL library, and shard it.<key,value> A key-value pair is a record. In this module, the key keyword represents the character offset of the URL string, and the value is the string information of the URL. Step 4, Map process: This process is to convert the<key,value> The keyword key and value value of the key-value pair are exchanged, and then the intermediate result is output. The keyword key represents the string information of the URL, and the value value is the character offset of the URL string. Step 5, Combiner process: merge the values ​​of the same keyword key in the intermediate files output in step 4 to improve the processing efficiency of subsequent operation steps; Step 6, Partitioner process: After the merging process in step 5, the intermediate results of the output of the Map task are split into several zones based on the key using a hash table. Each zone in the web crawling module corresponds to the same server node. Step 7, Reduce process: Based on the partitions in step 6, each Reduce task starts multiple download threads to crawl URLs in the same partition and outputs the crawled results; Step 8: Store in HDFS: Process the output of Reduce through OutputFormat and save it in the initial webpage library in the file system HDFS; Specifically, web page crawling is completed with the assistance of the tool software HttpClient. First, the crawling thread uses the HttpClient class to establish network communication with the web server, and applies the HTTP post or get function to make a connection request. Then, the web server responds to the request and returns the web page information in the form of a response message. Then, the web page message header information is checked, including the web page content length, character encoding set, content type, and redirection address, and the response message information is obtained through the getMethod.getResponseBody() method; finally, the corresponding web page content information is obtained by parsing the response message, and the content of the Robots file is read during the web page crawling process. Only those web page information that has permission or is allowed to be crawled are crawled, and the Robots protocol is followed. In addition, the parameter TIMEOUT is set during the process of establishing a connection with the web server. The TIMEOUT parameter is the maximum time allowed during the process of establishing a connection. If a link fails to establish a connection within the time exceeding TIMEOUT, the link will be discarded; Web page parsing module: It includes two aspects. On the one hand, it parses the captured web page to obtain the content information of the web page. On the other hand, it parses the hyperlink URL information contained in the web page. Before parsing the web page, the type of web page is identified according to the characteristics of different types of web pages. For different types of web pages, corresponding methods are selected to parse different types of web pages. The parsing process according to different types of web pages is as follows: Parsing HTML web pages: First, extract the web page content from the initial HTML file as the basis for subsequent processing. For HTML web pages, the primary task of the web page parsing module is to analyze the structure of the web page; The web page parsing module adopts the MapReduce cloud platform computing model. The Map() operation analyzes and filters various types of tags used for structured display of HTML type web pages, extracts the attributes and content of each tag, and stores the parsed content in the parsed web page library in the file system HDFS. Then, the Reduce() operation merges and summarizes all the link URLs extracted by the Map() operation and stores them in the link URL library in the file system HDFS for the spider to carry out subsequent crawling work. The description of the MapReduce model algorithm of the web page parsing module is as follows: Step 1: Initialization: Create a new MapReduce task and perform initialization operations on it; Step 2: Preprocessing: Preprocess the data through InputFormat and transform it into<key,value> This data format is key-value pairs, which is convenient for MapReduce tasks to process; Step 3, data sharding: pre-process the data of the initial webpage library, that is, obtain the initial information of the URL page from the initial webpage library, and perform sharding on it, and process the split data into<key,value> A key-value pair is a record. In this module, the key keyword represents the initial information of the page corresponding to the URL characters, and the value is the string information of the URL. Step 4, Map process: This process is to convert the<key,value> The keyword key and value value of the key-value pair are exchanged, and then the intermediate result is output. The keyword key represents the string information of the URL, and the value value is the initial information of the page corresponding to the URL characters; Step 5: Combiner process, that is, merging the values ​​of the same keyword key in the intermediate file output in step 4 to improve the processing efficiency of subsequent operation steps; Step 6, Partitioner process, after the merging process in step 5, splits the intermediate results of the Map task output into several zones based on the keyword key using a hash table. Step 7, Reduce process: Based on the partitioning in step 6, each Reduce task will parse the initial page information, parse the content information of the web page and the URL information of the link, calculate the comprehensive value of each URL link based on the improved web spider model algorithm, and output the crawled results; Step 8: Store in HDFS: Process the output of Reduce through OutputFormat and save it in the parsing web page library and parsing link library in the file system HDFS; Among them, the analysis of the HTML web page layout structure is assisted by the tool software HtmlParser. First, HtmlParser initializes all the content data in the web page, and then parses each tag separately, extracting the attributes and content of the Title, Body, and Link tags respectively. The parsed content will be stored in the parsed web page library in the file system HDFS, and the extracted web page link URL will be filtered and deduplicated by the link deduplication filtering module and then saved in the link outbound URL library in the file system HDFS for subsequent crawling work.

2. The web spider theme search system based on the improved cloud platform according to claim 1, characterized in that: Evaluate the comprehensive value of web pages: Based on the Shark-Search algorithm using the vector space model to calculate the topic relevance of web page information content, the algorithm is improved by adding the relevance information of the link structure. The calculation formula is Equation 10: Where sim(i) represents the topic relevance of web page i, fa(i) represents the set of all web pages linking to web page i, Inde(i) represents the number of backlinks of web page i, that is, the number of web pages in fa(i), the value range of variable q is between 0 and 1, and variable q adjusts the proportion of link structure and web page text, te_sim(i) is the similarity between web page content and topic, and the specific calculation is shown in formula 11: te_sim(i)=Sim(Q_Text(i), Th)×p+Sim(Q_Tit(i), Th)×(1-p) Equation 11 Q_Text(i) is the text content of web page i, Q_Tit(i) is the title information of web page i, Th represents the keyword set of the topic, and variable p is the same as q, its value range is between 0 and 1. Sim is calculated by using the TF-IDF algorithm based on the vector space model to calculate the feature vector of the document, and then the cosine theorem is used to calculate the similarity, as shown in Formula 9; Based on the calculation of the hyperlink structure value and the content topic relevance, the improved web spider model algorithm comprehensively considers the link value of web page q and the content topic relevance, and gives a calculation formula for calculating the comprehensive value of the web page, which is formula 12: Value(i)=(H[i]+A[i])×(s+sim(i)) Formula 12 Among them, Value(i) is the comprehensive value of the web page q to be finally obtained, H[i] and A[i] represent the Hub weight value and Authority weight value of web page i respectively, which are the results calculated by using Formula 4 and Formula 5 based on the HITS algorithm, sim(i) represents the topic relevance of web page i, which is the result calculated by using Formula 10; in addition, a control factor s is added to Formula 12, and its value range is between 0 and 1. A control factor is added to enable the web spider to crawl more topic-related pages.

3. The web spider theme search system based on the improved cloud platform according to claim 1, characterized in that: Based on the overall framework model of the system, the uncaptured link library, initial web page library, parsed link library, and parsed web page library are used: Uncrawled link library: The uncrawled URL library stores the currently pre-crawled URLs. After the URL segmentation module processes the URLs read from the URL seed file, they are stored in the file system HDFS. The web crawling module obtains URLs from the uncrawled URL library to perform specific web crawling work. The initial webpage library stores the initial webpage information captured by the webpage capture module from each node in the cloud platform web spider. After processing, they are saved in the file system HDFS for use by subsequent webpage parsing modules. The link analysis library stores the link analysis data obtained by the web page analysis module after the web page analysis module analyzes the initial web page information captured by each node in the cloud platform web spider. After processing, they are saved in the file system HDFS for subsequent crawling work; The web page parsing library stores the page parsing data obtained by the web page parsing module from the initial web page information captured by each node in the cloud platform web spider. After processing, the data is saved in the cloud platform file system HDFS for subsequent use. The above four repositories are implemented on the Hadoop-based file system HDFS to be called by the functional modules in the cloud platform web spider system.

4. The web spider theme search system based on the improved cloud platform according to claim 1, characterized in that: The system is divided into three parts: spider sub-nodes, master node, and HDFS. The master node coordinates the crawling between spider sub-nodes. Spider sub-nodes crawl pages and store the crawled pages in HDFS. The working mechanism of information and data interaction between the master node and the spider sub-node is: First, the master control node knows how many spiders are deployed in the entire system based on its own configuration information, and starts the spiders by remote login from the master control node; Second, after the spider child node is started, it sends a "Ready" status message to the main control node, indicating that the spider is ready to accept the crawling task; Third, after receiving the "Ready" message from the spider child node, the master control node generates an initial task, that is, encapsulating some URLs of the web pages to be crawled, and sending them to a certain path of the spider child node in the form of a file. Then, it sends a "Working" reply message to the spider node. If the master control node finds that there is no data in the current database, that is, there is no task to send, and there are still spiders crawling in the system, it will send a "Wait" reply message to the spider. If the spider system is started for the first time or all spiders are in the "Wait" state, it means that the user has not added a crawling task or the crawling is completed. The master control node will send a "Stop" message to each spider. Fourth, after receiving the "Working" reply message, the spider child node goes to the corresponding path and loads the seed URL into the crawling queue. If it is the first start, the crawling thread is started to start the crawling task. If it is not the first start, it continues to crawl the next task after the current crawling is completed. If the spider receives a "Wait" message, it sets the state to "Wait". After crawling the current task, it is in a waiting state until the master node wakes it up. If it receives a "Stop" message, the spider exits directly. Fifth, in the process of crawling pages, when the extracted URLs reach a certain amount, they will be encapsulated into a data file, that is, written into a text file, and the spider will send it to the path of the master node and send the status information of "Dada" to the master node; Sixth, after receiving the "Dada" message, the master control node loads the data into the database under the specified path; Seventh, when the spider is about to complete the crawling task, it will also send a "Ready" status message to the main control node, and then return to the first interactive state.

5. The web spider theme search system based on the improved cloud platform according to claim 1, characterized in that: The URL segmentation module segments the URLs in the seed URL set, which is the first step for the cloud platform web spider system to officially start; first, the URL segmentation module obtains the seed URL from the seed URL set; then, after completing the segmentation task, it distributes it to each Slave node for the crawling work of the web crawling module. The spiders on each Slave node store the distributed URL set in their respective uncrawled URL library, and then select a URL web link from the uncrawled URL library as the start of a specific crawling task. The most important task of the system is to ensure that the crawling tasks on each node do not conflict. At this time, task allocation is particularly important. Therefore, after the URL segmentation module segments the seed URL set, the spiders on each TaskTracker each obtain the URL set assigned to them. The spiders on each TaskTracker cooperate with each other and perform crawling tasks in parallel. The work of the URL segmentation module is to segment the URL list into several fragments and save them in the uncrawled URL library in the file system HDFS for the web crawling module to use. The MapReduce model algorithm of the URL segmentation module is described as follows: The first step is initialization: create a new MapReduce task and perform initialization operations on it; The second step is preprocessing: preprocess the data through InputFormat and transform it into<key,value> The key-value pairs in this data format are convenient for MapReduce tasks to process; The third step is data sharding: pre-processing the seed URL set, that is, obtaining URL link information from the seed URL set and sharding it; Step 4: Map process: perform associated operations on each URL according to the improved task allocation algorithm; Step 5, Combiner process: merge the values ​​of the same keyword key in the intermediate files output in step 4 to improve the processing efficiency of subsequent operation steps; Step 6, Partitioner process: After the merging process in step 5, the intermediate results of the output of the Map task are split into several regions based on the key using a hash table. Step 7, Reduce process: Based on the calculation results of the Map task, each Reduce task outputs the task allocation results of the URL links in the seed URL set; Step 8. Store in HDFS: Process the output results of Reduce through OutputFormat, and store the URLs assigned to each crawling sub-node in the uncrawled link library in its file system HDFS for the next crawling work.

Images