A network crawler scheduling method related to user behavior heat

Abstract

The application relates to a network crawler scheduling method associated with user behavior heat, and belongs to the technical field of network information acquisition. Network resource data is acquired through a network crawler, scheduling data is analyzed, the scheduling data comprises user preference behavior data for resources and resource historical update frequency data; user behavior heat values of the resources and resource update frequencies are respectively calculated according to the scheduling data, a resource update crawling queue is constructed on the basis, and the network crawler is scheduled according to the resource update crawling queue. The application firstly associates the user behavior heat for resources with the resource acquisition frequency, and schedules the network crawler according to the resource access heat, so as to provide the user with the resource collection effect closest to real-time update on one hand, and reduce the crawling frequency of non-popular resources, so as to reduce the information acquisition cost and save the network bandwidth on the other hand.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of network information acquisition, and particularly relates to a network crawler scheduling method associated with user behavior heat. BACKGROUND

[0002] A resource portal platform is a portal platform capable of effectively linking and collecting resources displayed by multiple resource platforms, such as search engine platforms, vertical business platforms, and the like. The resource portal platform (referred to as a portal platform) obtains a resource list of multiple resource platforms through a network crawler or an API interface and the like, and arranges and displays the resource list and abstract information by using a data analysis method. Since the network crawler is a main way of acquiring contemporary network resources, here, the way in which the portal platform acquires resources is collectively referred to as a "network crawler", including but not limited to a network crawler, an API interface, information subscription, and the like.

[0003] For the portal platform, the timeliness of resource acquisition will directly affect the user experience. Ideally, the resource list and abstract information displayed by the portal platform are consistent with the information displayed by the resource platform carrying the resource. However, since the resource can be changed at any time, the portal platform needs to repeatedly crawl to obtain updated information.

[0004] However, the number of network resources corresponding to the portal platform is huge, and frequent crawling will consume a large amount of system resources and network bandwidth. There are two facts, on the one hand, although the portal platform carries a large number of resources, the attention degree of the resources is quite different, and the proportion of resources with high access and high heat is not high; on the other hand, the update frequency of the resources is different, and generally, the resource update has a certain rule. Therefore, for the crawler system of the portal platform, effective scheduling can reduce the information acquisition cost and improve the user experience. SUMMARY

[0005] The present application aims at the deficiencies of the existing portal platform network crawler scheduling technology, and designs a network crawler scheduling method associated with user behavior heat. The user behavior heat of the resource is associated with the resource acquisition frequency for the first time, the network crawler is scheduled according to the resource heat, the resource collection effect closest to the real-time update can be provided for the user, the user experience is effectively improved, the network acquisition cost is considered, the crawling frequency of non-hot resources with infrequent update is reduced, the information acquisition cost is reduced, the system resources are saved, and the network bandwidth is saved.

[0006] In order to achieve the above purpose, the present application provides a network crawler scheduling method associated with user behavior heat, comprising the following steps:

[0007] Step A: acquiring network resource data through a network crawler respectively, and analyzing and scheduling data;

[0008] The network resource data is the network resource data displayed to the user by the resource portal type platform, and is used to update the resource content of the resource portal type platform;

[0009] The scheduling data includes user preference behavior data for network resources and network resource historical update frequency data, which are obtained by analysis during network crawler running;

[0010] Step B, calculating a user behavior heat value of the network resource according to the user preference behavior data for the network resource and a weight value of the user preference behavior for the network resource set to affect the popularity degree of the network resource;

[0011] Step C, predicting a network resource update frequency according to the network resource historical update frequency data;

[0012] Step D, constructing a resource update crawling queue, determining a crawling priority and a crawling frequency of each network resource according to the user behavior heat value of the network resource and the network resource update frequency;

[0013] Step E, performing network crawler scheduling according to the resource update crawling queue, and distributing network resource URLs to network crawler nodes according to the first-in first-out principle.

[0014] Further, the network resource data and the scheduling data are obtained by downloading, parsing web pages, and analyzing page update conditions through a crawling queue of network resources URLs formed by a network crawler.

[0015] Further, the user preference behavior for the network resource includes any one or more of browsing behavior, like behavior, downloading behavior, collection behavior, forwarding behavior, and comment behavior of the user for the network resource on the resource platform; for a network resource set R={r1, r2,..., r n}, the user preference behavior data for the network resource is a user preference behavior number set P={P1, P2,..., P n} of the user for the network resource on the resource platform in a preset period T, considering m kinds of user preference behaviors, the network resource r i corresponding P i ={p i1 , p i2 ,..., p im}, wherein p ik (k=1, 2,..., m) is the cumulative number of the user preference behavior k of the user for the network resource r i in the preset period T.

[0016] Further, in step B, a weight set W={w1, w2,..., w m, wherein the setting rule is that the greater the influence of the preference behavior k on the network resource popularity, the greater w k , and the following condition is met

[0017] The user behavior popularity of the network resource r i is obtained by , wherein the logarithm is base 2; the user behavior popularity threshold h t is set, and if h(r i ) < h t is met, the network resource r i is determined to be a non-popular network resource, and the user behavior popularity of the network resource is not limited to the above calculation method.

[0018] Further, the resource update crawling queue is a network resource URL queue waiting to be crawled by the network crawler, that is, a task queue of the network crawler, and the network resource URL includes two types of network resource URLs that are crawled for the first time and network resource URLs that are crawled at least once due to the need for updating.

[0019] Further, the construction of the resource update crawling queue comprises:

[0020] The network resource URLs are sorted according to the user behavior popularity value from large to small, and the queue construction principle is that the greater the user behavior popularity value of the network resource, the closer the URL is to the head of the queue, and the more times the URL appears in the queue, and each network resource URL appears at least once.

[0021] Further, the construction of the resource update crawling queue further comprises:

[0022] For a non-popular network resource, if the scheduled network resource crawling frequency is higher than the network resource update frequency predicted in step C, the number of times the network resource URL appears in the queue is reduced.

[0023] Further, steps A-E are performed according to a preset period T.

[0024] The present application provides a network crawler scheduling method associated with user behavior popularity, which has the following beneficial effects:

[0025] (1) The resource update crawling queue is constructed according to the user behavior popularity value of the network resource, which not only ensures that popular resources are crawled preferentially, but also increases the crawling frequency of popular resources, improves the timeliness of popular resources in the resource portal platform, and effectively improves the user experience, that is, provides the user with the effect of collecting resources closest to real-time updating;

[0026] (2) According to the resource history update frequency data to predict the resource update frequency, reduce the crawling frequency of the non-hot resource which is not frequently updated, reduce the unnecessary frequent crawling of the web crawler, improve the scheduling quality, reduce the information acquisition cost, and effectively save the system resources and save the network bandwidth. BRIEF DESCRIPTION OF DRAWINGS

[0027] In order to more clearly illustrate the technical solutions of the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings described below are some embodiments of the present application, and those skilled in the art can obtain other drawings according to these drawings without creative labor.

[0028] Figure 1 is a network crawler scheduling method flowchart associated with user behavior heat in the embodiment of the present application;

[0029] Figure 2 is a network crawler scheduling system schematic diagram associated with user behavior heat in the embodiment of the present application. DETAILED DESCRIPTION

[0030] In order to make the purpose, technical solutions and advantages of the present application more clear, the technical solutions of the present application will be described clearly and completely by embodiments with reference to the drawings of the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application. In each of the following embodiments, optional features and examples are provided, and each feature described in the embodiments can be combined to form multiple optional schemes, and each numbered embodiment should not be regarded as only one technical solution.

[0031] The present application will be further described and specifically explained in detail with reference to the drawings and embodiments.

[0032] Embodiment one

[0033] Figure 1 is a network crawler scheduling method flowchart associated with user behavior heat provided by the first embodiment of the present application, and the method comprises:

[0034] Step A, obtaining network resource data through a network crawler respectively, and analyzing and scheduling data;

[0035] The network resource data is the network resource data displayed to the user by the resource portal type platform, and is used to update the resource content of the resource portal type platform;

[0036] The scheduling data includes user preference behavior data for network resources and historical update frequency data of network resources, which are obtained through analysis during the operation of the web crawler.

[0037] Step B: Calculate the user behavior popularity value of the network resource based on the user's preference behavior data for network resources and the weight value set for the influence of user preference behavior on the popularity of network resources.

[0038] Step C: Predict the network resource update frequency based on the historical update frequency data of the network resources;

[0039] Step D: Based on the user behavior popularity value of the network resources and the update frequency of the network resources, construct a resource update crawling queue and determine the crawling priority and crawling frequency of each network resource.

[0040] Step E: Schedule web crawlers according to the resource update crawling queue, and distribute web resource URLs to web crawler nodes according to the first-in-first-out principle.

[0041] When crawling the resource platform for the first time, the web crawler is scheduled according to the initial resource crawling queue. This queue contains the URLs of network resources waiting to be crawled, and each resource URL appears once.

[0042] The network resource data and scheduling data are obtained by web crawlers downloading, parsing web pages, and analyzing page update status based on a crawling queue composed of network resource URLs.

[0043] User preferences for online resources include any one or more of the following behaviors on the resource platform: browsing, liking, downloading, saving, forwarding, and commenting; for the set of online resources R = {r1, r2, ..., r...} n The user's preference behavior data for network resources is a set P = {P1, P2, ..., P} of the number of times a user exhibits preference behavior for network resources on the resource platform within a preset period T. n Considering m types of user preference behaviors, and network resources r i Corresponding P i ={p i1 p i2 , ..., p im}, where p ik (k = 1, 2, ..., m) represents the user's request for network resource r. i The cumulative number of preference behaviors k in a preset period T.

[0044] In step B, a set of influence weights W = {w1, w2, ..., w...} is pre-defined. m The rule is set such that the greater the influence of preference behavior k on the popularity of network resources, the higher w becomes.k The larger, and satisfying

[0045] Network resources r i User behavior heat index We can obtain that the logarithm is base 2; set the user behavior popularity threshold h. t If h(r) is satisfied i )<h t Then determine the network resource r i For non-popular online resources, the user behavior popularity of online resources is not limited to the above calculation methods.

[0046] In step D, the resource update crawling queue is a queue of network resource URLs waiting to be crawled by the web crawler, i.e., the web crawler's task queue. The network resource URLs include two types: those being crawled for the first time and those that have been crawled at least once due to update needs. The specific queue construction steps include:

[0047] First, according to resource r i User behavior heat value h(r) i Sort the resources in descending order of size, following the order h(r). i The larger the value of r, the more resources are available. i The closer the URL is to the head of the queue, the more times it appears in the queue (c). i Construct a resource queue q based on the principle of larger values, satisfying c. i ≥1, thus ensuring both priority crawling of popular resources and increasing the crawling frequency of popular resources; secondly, when resource r i When the resource is not in high demand, the resource update frequency f predicted in step C is... i If satisfied If the resource crawling frequency is higher than the resource update frequency at this time, then modify... Where max is the function for finding the maximum value. This is the floor function.

[0048] All steps AE are performed according to a preset period T.

[0049] Example 2

[0050] This embodiment, based on Embodiment 1 above, further elaborates on the method for predicting resource update frequency. This embodiment shares the same inventive concept as the above embodiments, and the same or corresponding terminology will not be repeated here.

[0051] Different types of resource platforms have different publishing modes, that is, the resource updates thereof are subject to different rules, and therefore it is necessary to mine the resource update distribution rules according to the historical update frequency data of the resources so as to predict the resource update frequency. Here, the news website is taken as an example for illustration:

[0052] According to the statistics and experiments on the resource publishing data of the news website, it is verified that the resource update process thereof conforms to the Poisson distribution. Thus, it is assumed that the number of accesses to a resource r s is n i in a period of time T i , and the number of updates of the resource in the n i accesses is x i . According to the Poisson distribution, the update frequency of the resource r i can be estimated as where f i represents the expected value of the number of occurrences of an event per unit time.

[0053] Embodiment Three

[0054] This embodiment is further illustrated on the basis of the above-described embodiment one for the case where the resource has no historical update frequency data. This embodiment has the same inventive concept as the above-described embodiment, and the same or corresponding terms are explained as in the above-described embodiment, and this embodiment will not be described herein again.

[0055] When the resource r i is first crawled by the web crawler, or the resource r i has no update since the first crawling, the historical update frequency data of the resource cannot be obtained, and the update frequency of the resource is set as , that is, the update period of the resource is assumed to be the same as the preset period T.

[0056] Embodiment Four

[0057] This embodiment is further illustrated on the basis of the above-described embodiment one for the case where the user behavior heat values of all resources are equal. This embodiment has the same inventive concept as the above-described embodiment, and the same or corresponding terms are explained as in the above-described embodiment, and this embodiment will not be described herein again.

[0058] If the user behavior heat values of all resources are equal at the current T, that is, h(r i ) = h(i = 1, 2,..., n), where h is a constant, it indicates that the user behavior heat of each resource has no distinguishability, and the user behavior cannot guide the crawler scheduling. In this case, the resource update crawling queue is constructed according to the original order of the resources of the target resource set R = {r1, r2,..., r n}, that is, the crawler scheduling is returned to the ordinary crawler scheduling which is not associated with the user behavior heat.

[0059] Embodiment five

[0060] Figure 2 is a network crawler scheduling system provided by embodiment five of the present application, which is a schematic diagram of associating user behavior heat, comprising:

[0061] a calculation module for operating the scheduling data in the database;

[0062] The calculation module comprises a heat calculation unit for calculating the user behavior heat value of the network resource according to the preference behavior data of the user for the network resource and the set weight value; a frequency calculation unit for predicting the resource update frequency according to the historical update frequency data of the network resource;

[0063] a scheduling module for constructing a resource update crawling queue, determining the crawling priority and crawling frequency of each network resource according to the user behavior heat value and the network resource update frequency of the network resource output by the calculation module; then performing network crawler scheduling according to the resource update crawling queue, and distributing the network resource URL to the network crawler node according to the first-in first-out principle;

[0064] a crawler node for initiating a network request to the server of the resource platform according to the network resource URL distributed by the scheduling module, obtaining the response containing the updated resource related data returned by the opposite server, and delivering it to the parsing module;

[0065] a parsing module for parsing the response according to certain rules, extracting the network resource data and scheduling data, and storing them in the database;

[0066] After the crawler node stores the network resource data and the scheduling data in the database, the network resource data in the database is the resource data updated by the synchronization resource platform, which is used for the collection, linking and display of the resource portal type platform; the scheduling data in the database is delivered to the calculation module, and after being calculated by the calculation module, it guides the network crawler scheduling.

[0067] The above is the preferred embodiment of the present application, and the present application should not be limited to the content disclosed in the embodiment and the drawings. Any equivalent or modification without departing from the disclosed completion falls within the scope of protection of the present application.

Claims

1. A method for scheduling a web crawler in association with a user behavior heat, characterized in that, The method comprises: Step A, obtaining network resource data respectively through a network crawler, and analyzing scheduling data; The network resource data is network resource data displayed to a user by a resource portal type platform, and is used to update resource content of the resource portal type platform; The scheduling data comprises user preference behavior data for network resources and network resource historical update frequency data, and is obtained by analysis during network crawler operation; Step B, calculating a user behavior heat value of the network resource according to the user preference behavior data for network resources and a weight value of a user preference behavior for network resources that affects a network resource popularity degree set; The weight values are set by pre-setting a weight set W = {w1, w2, …, wm} where the setting rule is that the greater the influence of the preference behavior k on the network resource popularity, the greater w m , k and 0 < w j < 1 (j = 1, 2, …, m), The network resource r i The user behavior hotness of the network resource r is calculated, where the logarithm is base 2; a user behavior hotness threshold h t is set, and if h(r i ) < h t is satisfied, the network resource r i is determined to be a non-hot network resource. Step C, predicting a network resource update frequency according to the network resource historical update frequency data; Step D, constructing a resource update crawling queue according to the user behavior heat value of the network resource and the network resource update frequency, and determining a crawling priority and a crawling frequency of each network resource; Step E, performing network crawler scheduling according to the resource update crawling queue, and distributing network resource URLs to network crawler nodes according to a first-in first-out principle. 2.The network crawler scheduling method of claim 1, wherein, The network resource data and the scheduling data are obtained by downloading, parsing a webpage, and analyzing a page update situation through a network crawler according to a crawling queue of network resource URLs.

3. The method of claim 1, wherein the method further comprises: The preference behaviors of the user to the network resources include any one or more of browsing behavior, like behavior, downloading behavior, collection behavior, forwarding behavior and comment behavior of the user to the network resources on the resource platform; for a network resource set R={r1, r2,..., r n}, the preference behavior data of the user to the network resources is a set of times P={P1, P2,..., P n} of the preference behaviors of the user to the network resources on the resource platform in a preset period T, considering m kinds of user preference behaviors, the network resource r i corresponding P i ={p i1 , p i2 ,..., p im}, wherein p ik (k=1, 2,..., m) is the cumulative number of the preference behavior k of the user to the network resource r i in the preset period T.

4. The method of claim 1, wherein, The resource update crawling queue is a network resource URL queue waiting to be crawled by the network crawler, that is, a task queue of the network crawler, and the network resource URLs comprise two types of network resource URLs that are crawled for the first time and network resource URLs that are crawled at least once due to update needs.

5. The method of claim 1, wherein, The construction of the resource update crawling queue comprises: sorting the network resource URLs according to the user behavior heat values from large to small, and the queue construction principle is that the greater the user behavior heat value of a network resource is, the closer the URL is to the head of the queue, and the more times the URL appears in the queue, and each network resource URL appears at least once.

6. The method of claim 1, wherein, The construction of the resource update crawling queue further comprises: for non-popular network resources, if the scheduled network resource crawling frequency is higher than the network resource update frequency predicted in step C, the number of times of appearance of the network resource URL in the queue is reduced.

7. The method of claim 1-6, wherein, Steps A-E are performed according to a preset period P.

Images