Efficient freshness crawl scheduling

Abstract

The techniques described herein construct an optimal refresh schedule by minimizing a cost function constrained by available refresh bandwidth. The cost function receives as input an importance score for a content item and a rate of change for the content item in order to optimize the schedule. The cost function is considered optimized when a refresh schedule is found that no longer minimizes the cost while using the available bandwidth. The techniques are able to construct an optimized schedule to refresh content with incomplete change data, content with complete change data, or a mix of content with and without complete change data. It is also able to relearn the rate of change of a content item from its own scheduling history and recalculate the refresh schedule, ensuring that the schedule takes into account the latest trends in content item updates.

Description

BACKGROUND

[0001] Web crawlers are a typical part of search engines that obtain information that is then provided to users of the search service. As the Web becomes more dynamic, in addition to discovering new web pages, crawlers need to constantly revisit those web pages that are already in the search engine index in order to keep the index fresh by picking up the changed content of the pages. This refreshing process is resource intensive. SUMMARY

[0002] This summary is provided to introduce a selection of concepts, in a simplified form, that are further described below in the detailed description. This summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to determine the scope of the claimed subject matter.

[0003] The technology described herein provides more efficient crawling schedules for content trackers, such as search engines, to use when refreshing a content index. As the Web becomes more dynamic, services that rely on Web data are facing the problem of keeping up with content changes that are increasingly challenging. Whether it is a continuous query system, a virtual assistant, or an Internet search engine, such a service tracks many remote content sources - web pages or data streams. The user expects the tracker to reveal the latest content at the source now. For all the content sources that the tracker monitors, the tracker decides when to re-request (crawl) the content source in order to refresh the content given the changes made since the last time the content was crawled. The strategy of making these crawling decisions well addresses the freshness crawling scheduling problem.

[0004] The technology described herein is an overall approach to freshness crawling scheduling that is computationally efficient and uses a type of reinforcement learning to generate optimal freshness results. In an ideal world, a search engine would revisit a content source as soon as the content changes. As a practical matter, this is typically not possible technically: most content sources (e.g., web pages) do not notify the search engine when their content changes. Even if they did, network bandwidth constraints would prevent the crawler of the search engine from updating the content at every change notification. The goal is to find the optimal time to revisit these content sources, taking into account the overall available bandwidth as well as the estimated rate of change of the content source, the importance of the content source, and the received change notifications.

[0005] At a high level, the techniques described herein construct an optimal refresh schedule by minimizing a cost function that characterizes the penalty incurred by a search engine due to staleness (i.e., lack of freshness of its index) subject to the constraint of available refresh bandwidth. The cost function takes as input the importance scores of the content items defined by the search engine and the rate of change of the content items in order to optimize the schedule. The optimization problem is solved when a refresh schedule is found that no longer minimizes the cost while using the available bandwidth. The optimal refresh schedule (sometimes described herein as a refresh policy) assigns an amount of the available bandwidth to each content item in the set to be refreshed. In total, the bandwidth assigned to each item within the set is less than or equal to the available bandwidth.

[0006] The optimal schedule created using the techniques described herein will maximize the overall item freshness. Both the rate of change of the content items and the importance scores for the content items are used to assign bandwidth. The techniques described herein are further characterized in that, over the time horizon covered by the schedule, all content items in the set to be refreshed will be assigned at least some bandwidth.

[0007] The techniques described herein can be optimized to construct a refresh schedule for content items that have different amounts of associated change data. The techniques can construct an optimal schedule to refresh content items that have incomplete change data (when the search engine observes the item from time to time and thereby detects changes, but does not know how many changes occurred between the observed values), content items that have complete change data (when the search engine is notified every time a content item changes), or a mix of content items that have and do not have complete change data. The optimal schedule is constructed by choosing a differently parameterized cost function for different scenarios. In particular, the cost function for a site that has complete change data can depend on the probability of crawling in response to a change notification for the content site. In other words, the system can not crawl every content item when a change notification is received. Instead, a probability is used to determine whether to visit the site, with different probability values translating to different values of the cost function. Conceptually, this can be imagined as flipping a coin to determine whether to visit the site when a change notification is received. Instead of the 50% probability associated with flipping a coin, the probability computed for the item will be used to make the decision to crawl or not to crawl. BRIEF DESCRIPTION OF DRAWINGS

[0008] The present disclosure is described in detail below with reference to the attached drawing figures, wherein:

[0009] Figure 1 is a block diagram illustrating a general process of refreshing a content index according to various aspects of the techniques described herein;

[0010] Figure 2 is a block diagram of an exemplary computing environment suitable for use with various aspects of the techniques described herein;

[0011] Figure 3 FIG. 1 is a block diagram illustrating an example content refresh system in accordance with various aspects of the techniques described herein;

[0012] Figure 4 FIG. 2 is a flow diagram illustrating a method for preparing a crawl probability vector for a crawler to follow when seeking an updated copy of a content item in accordance with various aspects of the techniques described herein;

[0013] Figure 5 FIG. 3 is a flow diagram illustrating a method for preparing a crawl probability vector for a crawler to follow when seeking an updated copy of a content item in accordance with various aspects of the techniques described herein;

[0014] Figure 6 FIG. 4 is a flow diagram illustrating a method for preparing a crawl probability vector for a crawler to follow when seeking an updated copy of a content item in accordance with various aspects of the techniques described herein;

[0015] Figure 7 FIG. 5 is a block diagram illustrating an example computing device; and

[0016] Figure 8 FIG. 6 is a flow diagram illustrating a method for preparing a crawl probability vector for a crawler to follow when seeking an updated copy of a content item in accordance with various aspects of the techniques described herein. DETAILED DESCRIPTION

[0017] The subject matter of the present disclosure is now described with reference to specific embodiments. The descriptions and examples are intended to convey an understanding of the subject matter and are not intended to be limiting. Conjunction with other prior or future technologies, the inventors have contemplated that the claimed subject matter can also be implemented in other ways to include different steps or combinations of steps similar to those described in this document to include other cost functions. Also, although the terms “step” and / or “block” can be used herein to connote different elements of the methods employed, the terms should not be interpreted as implying any particular order among or between various steps herein disclosed unless and except when the order of individual steps is explicitly described and nothing else. Each method described herein can comprise a computer process implemented using hardware, firmware, and / or software. For example, a variety of functions can be performed by a processor executing instructions stored in memory. These methods can also be implemented as computer-usable instructions stored on computer storage media (e.g., computer-readable media). These methods can be provided using an independent application, a service hosted by a server, a service hosted by a cloud computing provider, or a plug-in to another product, to name a few.

[0018] The technology described herein provides more efficient crawl scheduling for content trackers, such as search engines, to use when refreshing content indexes. As the Web becomes more dynamic, services that rely on Web data are facing the problem of keeping up with content changes more and more challenging. Whether it is a continuous query system, a virtual assistant, or an Internet search engine, such a service tracks many remote content sources—web pages or data streams. The user expects the tracker to reveal the latest content at the source now. For all the content sources that the tracker monitors, the tracker decides when to re-request (crawl) the content source in order to refresh the content given the changes made since the last time the content was crawled. The strategy of making these crawl decisions well addresses the freshness crawl scheduling problem.

[0019] Freshness crawl scheduling has multiple challenging aspects. For most content sources, the tracker can only discover whether the content has changed when it crawls the source. To guess when changes occur and should thus be downloaded, the tracker needs prediction models whose parameters are initially unknown. Therefore, the tracker needs to learn these models and optimize freshness-related objectives when scheduling crawls. However, for some web pages, sitemap polling and other means can provide trustworthy near-immediate signals that a page has changed in a meaningful way, even though the change is not exact. But even with these remote change observations and known change model parameters, freshness crawl scheduling is still very important because the tracker cannot react to every single predicted or actual change. The tracker's infrastructure imposes bandwidth constraints on the average number of daily crawls, often limiting it to a small fraction of the amount of change events.

[0020] The technology described herein is an overall approach to freshness crawl scheduling that is computationally efficient and uses a type of reinforcement learning to generate optimal freshness results. In an ideal world, a search engine would re-visit a content source as soon as the content changes. As a practical matter, this is often technically impossible: most content sources (e.g., web pages) do not notify a search engine when their content changes. Even if they did, network bandwidth constraints would prevent the search engine's crawler from updating the content at every change notification. The goal is to find the optimal time to re-visit these content sources, taking into account the overall available bandwidth as well as the estimated change rate of the content source, the importance of the content source, and the received change notifications.

[0021] At a high level, the techniques described herein construct an optimal refresh schedule by minimizing a cost function that penalizes a search engine for staleness (i.e., a lack of freshness of its index) constrained by available refresh bandwidth. The cost function takes as input a search engine-defined importance score of content items and a rate of change of content items in order to optimize the schedule. The optimization problem is solved when a refresh schedule is found that no longer minimizes the cost while using the available bandwidth. The optimal refresh schedule (sometimes described herein as a refresh policy) assigns an amount of available bandwidth to each content item in a set to be refreshed. In total, the bandwidth assigned to each item within the set is less than or equal to the available bandwidth.

[0022] The optimal schedule created using the techniques described herein will maximize overall item freshness. The techniques described herein are further characterized in that, over a time horizon covered by the schedule, all content items in the set to be refreshed will be assigned at least some bandwidth. As will be described in more detail, individual content items can not be crawled over the time horizon, however, the plan allows for the possibility that content items can be crawled depending on events, such as receiving a notification of a change to a content item that occurs during the time horizon.

[0023] A key goal of the techniques described herein is to generate an optimal refresh schedule. As used herein, an optimal refresh schedule maximizes overall freshness of content items within a set to be refreshed. Further, the importance of items is taken into account when assigning bandwidth. An importance score is used to define the importance of an item. As used herein, an item is stale when it has changed and has not been crawled by the refresh process. Otherwise, if the item has not changed since it was last downloaded to the cache, the item is fresh.

[0024] In contrast to non-optimal approaches, the optimization can be better understood, such as distributing crawl bandwidth evenly across all items in a set. Distributing crawl resources evenly across items in a content set is not optimal when considering only the rate of change, as frequently changing sites are crawled as often as infrequently changing sites. Given a bandwidth-limited environment, this means that frequently changing sites can be stale for a relatively long time, while infrequently changing sites will be crawled multiple times before a change occurs. Thus, evenly distributing crawl opportunities does not maximize overall freshness. Other simple heuristics, such as distributing crawl opportunities in proportion to only the rate of change or importance score, also yield suboptimal results.

[0025] The more times a content item has changed since it was last refreshed / crawled by the tracker, the higher per-unit-time penalty the cost function used with the techniques described herein assigns to the content item. Thus, it penalizes the tracker for every change that has occurred at the source since the previous crawl, not just the first time. This can be described as a monotonically increasing penalty. In one aspect, the cost increase is not linear. Rather, the cost function is chosen so that as the number of unsynchronized changes at the content item increases, the cost increases at a decreasing rate. This reflects an intuition that while all undetected changes at a content item are important, the first is most important because it marks the transition from fresh to stale. Thus, in one aspect, the cost function related to the rate of change is both monotonically increasing and discretely concave, providing a decreasing increase in penalty over time.

[0026] The importance score is also used in the optimization schedule. The importance score assigned to a content item can take many forms and can be constructed with a variety of features as inputs. For example, a spam score can be an input to the importance score. The web traffic of a content item can be an input to the importance score, with sites having high traffic being more important. The click-through rate of search results can be another factor, with high click-through rates leading to higher importance scores. Similarly, higher query frequency can lead to higher importance scores. In one aspect, various content items already have importance scores assigned by search engines or other services (even if not called importance scores), and that score can be used by aspects of the techniques as the importance score or as a component of a new score. In one approach, different factors are given different weights to produce the importance score. In one aspect, the importance score is greater than zero because using zero within the cost function would result in zero cost for not accessing the content item. Zero cost can result in zero scheduled crawls of the item, which is not consistent with the goals of the techniques described herein. If a content item should not be refreshed, it is better to exclude it from the set being evaluated.

[0027] The goal of the techniques described herein is to construct a refresh schedule that does not starve any content item in the set to be refreshed. In other words, every item in the set should have a chance to be crawled during the scheduled time covered by the refresh schedule. As described in more detail, the cost function takes the rate of change and the importance score as inputs. Aspects of the techniques ensure that the refresh solution does not result in any content item having a chance of not being crawled during the scheduled range.

[0028] In some cases, it can be desirable to exclude content items from refresh. For example, known spam pages can have zero importance and can not be desirable to refresh. Similarly, sites known not to change can also be excluded. This can be handled by not including the site in the set of content items to be refreshed. As used herein, the set of content items need not include all known content items in the cache or index.

[0029] The techniques described herein can be optimized to build refresh schedules for content items with different amounts of associated change data. The techniques can build optimized schedules to refresh content items with incomplete change data (when a search engine observes an item from time to time and thereby detects changes, but does not know how many changes the item makes between observations), content items with complete change data (when the search engine is notified every time a content item changes), or a mix of content items with and without complete change data. The optimized schedules are built by choosing different parameterized cost functions for different scenarios. In particular, the cost function for a site with complete change data can depend on the probability of crawling in response to a change notification for the content site. In other words, the system can not crawl every content item when a change notification is received. Instead, a probability is used to determine whether to visit the site, with different probability values translating to different values of the cost function. Conceptually, this can be imagined as flipping a coin when a change notification is received to determine whether to visit the site. Instead of the 50% probability associated with flipping a coin, the probability computed for the item will be used to make the decision to crawl or not to crawl. For example, a random number generator can be used to generate a number between 1 and 100 when a change notification for a content item is received. When the randomly generated number is less than or equal to 60, a crawl probability of 0.6 can be achieved by crawling the content item.

[0030] Content items have complete change data when change notifications are received nearly instantaneously every time the content item changes. Change data can be obtained by leveraging telemetry signals from users accessing different content. For example, a web browser can provide web page information to a change analysis component of a tracker. The web browser can be associated with the tracker. Frequently visited websites can have complete change data provided by the browsers of many visitors. The website itself can maintain site map files that reliably record every change on the website pages and can be polled very frequently by the tracker to find the changes. A threshold can be used to determine what level of change information is needed to classify an item as having complete change information. The threshold can take into account the frequency of analysis and the historical frequency of changes. Any content item that does not meet the complete change threshold is classified as incomplete.

[0031] All of these topics will be explained in more detail later, particularly with reference to Figures 3 to 6 .Figure 1 An introduction to the concept of crawling and how refresh scheduling can be used is provided, and Figure 2 A suitable computing environment for the technology is described.

[0032] Figure 1 An exemplary scenario 100 is presented that generally illustrates these and other scenarios involving a collection of cache items 104 stored in a project cache 102. The cache items 104 correspond to content items 108 retrieved from one or more content item hosts 106 and are stored in the project cache 102 to provide fast access to frequently requested sets of content, e.g., in response to queries. For example, these queries can be specific requests for a content item 108 hosted by the content item host 106, such as a URL identifying a particular web page or web resource, or can be more general queries that produce a set of search results describing relevant content items 108. The result sets can be stored as items 104 in the project cache 102 and can be frequently updated based on changes to the content items 108 so queried.

[0033] As the content items 108 provided by the content item hosts 106 change over time, the items 104 in the project cache 102 must be refreshed to avoid producing stale results that reference old versions of the content items 108. Accordingly, a set of cache refresh resources 120 can be used to refresh the items 104 in the project cache 102. For example, the set of cache refresh resources 120 can include a first refresh resource 122 and a second refresh resource 124, each of which can include a process designed to refresh the items 104 in the project cache 102. As one possible, but inefficient example, the cache of refresh resources 122 can be configured to randomly select an item 104 from the project cache 102, retrieve the corresponding content item 108 from the corresponding content item host 106, and compare the item 104 to the content item 108. If any discrepancies are detected between the item 104 and the content item 108, the refresh process can write the content item 108 over the stale version 108 of the content item 108 stored in the project cache 102, thereby refreshing the item 104. This can be described as a crawl event. The time at which the crawl event occurred and whether the change was detected in the content can be recorded. This data can be used to determine the rate of change of the content, as described subsequently.

[0034] In these and other scenarios, while not necessarily fatal to the operation of item cache 102, staleness will be avoided by allocating cache refresh resource set 120 in a manner that reduces the probability of serving outdated items. For example, cache refresh resource set 120 can be allocated to refresh items 104 in item cache 102 based on various relevant factors. In one aspect, first cache refresh resource 122 can be responsible for refreshing the set of content items with limited real-time changed data. Second cache refresh resource 124 can be responsible for refreshing the set of content items with complete real-time changed data. Each of the first cache refresh resource 122 and the second cache refresh resource 124 can have a specified amount of crawl bandwidth.

[0035] Cache refresh strategies can be chosen to pursue a number of priorities, such as allocating resources to refresh the most frequently requested item 104, maximizing the percentage of fresh items 104 in item cache 102, or most effectively allocating cache refresh resource set 120 so that requests are only issued for content items 108 that are most likely to have been updated since the last refresh of the corresponding item 104. Depending on the selected priorities and their relative importance, the strategy for refreshing item cache 102 can be configured to allocate cache refresh resource set 120 that most effectively promotes these priorities, i.e., to achieve the most useful allocation of cache refresh resource set 120. Therefore, refresh strategies used in these techniques can involve using the predicted query frequency and update frequency of the corresponding item 104 to design a refresh strategy that achieves high refresh utility based on the selected set of priorities and their relative importance.

[0036] Now go to Figure 2 A block diagram is provided illustrating an operating environment 200 in which some embodiments of this disclosure may be employed. It should be understood that such and other arrangements described herein are merely illustrative. Other arrangements and elements (e.g., machines, interfaces, objective / cost functions, sequences, and functional groupings, etc.) can be used in addition to or instead of the arrangements shown, and some elements may be omitted entirely for clarity. Furthermore, many of the elements described herein are functional entities that can be implemented as discrete or distributed components or in combination with other components, and implemented in any suitable combination and in any suitable location. The various functions described herein as being performed by one or more entities can be performed by hardware, firmware, and / or software. For example, some functions can be performed by a processor executing instructions stored in memory.

[0037] Among other components not shown, the operating environment 200 includes: multiple user devices, such as user devices 202a and 202b to 202n; multiple content sources, such as content sources 204a and 204b to 204n; a search server 206; a cache 205; a user 203a; and a network 220. It should be understood that... Figure 2 The environment 200 shown is an example of a suitable operating environment. For example, Figure 2 Each component shown can be implemented via any type of computing device, such as a combination Figure 7 The computing device 700 is described. These components can communicate with each other via a network 220, which may include, but is not limited to, one or more local area networks (LANs) and / or wide area networks (WANs). In an exemplary implementation, network 220 includes the Internet and / or a cellular network in any of a variety of possible public and / or private networks.

[0038] It should be understood that, within the scope of this disclosure, any number of user devices, servers, users, and content sources can be employed within operating environment 200. Each device can include a single device or multiple devices cooperating in a distributed environment. For example, search server 206 can be provided via multiple devices arranged in a distributed environment that collectively provides the functionality described herein. Additionally, other components, not shown, may also be included within the distributed environment.

[0039] User equipment 202a and 202b to 202n may include any type of computing device capable of being used by user 203a. For example, in one embodiment, user equipment 202a to 202n may be, as described herein, [the type of computing device]. Figure 7 The type of computing device described. By way of example and not limitation, user equipment can be implemented as a personal computer (PC), laptop computer, mobile device, smartphone, tablet computer, smartwatch, wearable computer, virtual reality headset, augmented reality headset, personal digital assistant (PDA), MP3 player, global positioning system (GPS) or device, video player, handheld communication device, gaming device or system, entertainment system, in-vehicle computer system, embedded system controller, camera, remote control, barcode scanner, computerized measuring device, electrical appliance, consumer electronic device, workstation, or any combination of these described devices or any other suitable device.

[0040] User devices 202a and 202b through 202n can be client devices on the client side of operating environment 200, while search server 206 can be on the server side of operating environment 200. Each user device 202a and 202b through 202n can be associated with one or more users (such as user 203a). Some user devices can be associated with multiple users, such as home PCs, game consoles, conference room PCs, electronic whiteboards, etc. Similarly, a single user can be associated with multiple devices, including shared devices. User login identifiers can be used to identify the user operating the user device at a given time and associate the actions taken with the user record. User devices can submit queries and provide webpage summaries that can be used to detect changes when navigating to content items.

[0041] Content sources 204a and 204b to 204n may include content sources and / or content systems configured to make content available for operating environment 200 or in conjunction with it. Figure 3 Any of the various components of the refresh scheduling environment 300 described. Content sources 204a and 204b through 204n can be web hosts. Search server 206 can store copies of content from the content sources in cache 205.

[0042] Now go to Figure 3 A block diagram is provided illustrating an exemplary refresh scheduling environment 300 from which some embodiments of the present disclosure may be employed. The refresh scheduling environment 300 includes, in conjunction with... Figure 2 A network 220 describes and communicatively couples components of a refresh scheduling environment 300. For example, components of the refresh scheduling environment 300 may be implemented as compiled computer instructions or functions, program modules, computer software services, or in one or more computer systems (such as those combined with...). Figure 7 The set of arrangements of processes executed on the described computing device 700.

[0043] The refresh scheduling environment 300 comprises content sources 204a and 204b to 204n, user device 320, and refresh system 330. The user device includes a browser 322, which includes the ability to send change information to a change notification component 332. The refresh system 330 includes a change notification component 332, a change frequency determiner 334, an importance score calculation component 336, an incomplete change data scheduler 338, a complete change data scheduler 340, a hybrid scheduler 342, a crawler 344, and a content cache 346. Furthermore, Figure 3These components, the functions performed by these components, or the services performed by these components can be implemented at multiple appropriate abstraction layers (such as operating system layer, application layer, hardware layer, etc.) of the computing system(s). Alternatively or additionally, the functionality of these components and / or the embodiments of this disclosure described herein can be performed at least in part by one or more hardware logic components. For example, and not as a limitation, illustrative types of hardware logic components that can be used include field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), etc. Additionally, although the functionality is described herein with respect to the specific components shown in the refresh scheduling environment 300, it is contemplated that in some embodiments, the functionality of these components can be shared or distributed across other components.

[0044] The change notification component 332 identifies content changes by analyzing signals received from client devices or other sources. Besides crawling, some webpage changes can also be detected. For example, browsers can capture a fingerprint of content and provide this to search engines. In this way, search engines receive the fingerprint stream. The fingerprint can be a hash of the page content. If the page hasn't changed, the fingerprint will remain the same. On the other hand, if the page has changed, the fingerprint will also change. The search engine will then know that the page has changed.

[0045] Browser (or other client) functionality can provide a webpage summary to a content change component, which analyzes the summary to determine if any meaningful changes have occurred. The browser function detects HTML sections of the webpage, calculates a summary (such as a hash of each section), and sends information about the webpage to the change component. The submitted information may include: section hashes, the webpage's URL, and the date and time of access. If applicable, data from the "last modified" HTML tags may be provided. The summary may include information from the following parts of the webpage: title, header, body, and footer. The hash is then compared to a previous hash of the page / page section. Changes in the hash reflect changes to the page.

[0046] In one aspect, a web host or other content source can provide content update notifications. It is generally in the interest of the content provider to have the latest content on the search engine. Upon delivery, this notification is considered a change confirmation. When a change is detected, a change notification can be passed to the change notification component 332. A record of changes can be saved and used to calculate the frequency of changes to the item.

[0047] The change frequency determiner 334 calculates the estimated change rate for individual content items. The estimated change rate can be computed using a reinforcement learning method. This method updates the estimated change rate using change observations obtained by the crawling source based on a refresh schedule derived from the current change rate estimate. While staleness minimization is effective under the model described in this paper, in reality, the actual change rate is often unavailable and changes over time, requiring frequent relearning. Improved change rate estimation can lead to improved refresh scheduling.

[0048] Suppose that for a given source w, the tracker has observed binary change indicator variables z1, z2, ..., z... U The sequence, where t0, ..., t U It is the time of observation, and for 1≤j≤U, if and only if with time t j-1 Compared to the source changing at least once, z j =1. Consider how changes can be detected when complete change information is available and incomplete change information is available.

[0049] When incomplete change information is available for w, the tracker crawls the content items to determine whether individual content items have changed between crawling events. At times t1, ..., t... U After each crawl, the tracker checks for changes compared to previous crawls of the content items, and if the content differs, it records z. j =1, or if the content is the same, record z. j =0. Changes can be determined by looking at the current content and / or structure of a content item in light of previous versions. For example, text can be compared by isolating the text within a content item, counting the total characters in the text, and counting the distinct characters in the text. When a change in the total number of characters or the distinct characters occurs between different accesses to the content item, then the content item is determined to have changed. There may be more than one change since the last crawl, but the tracker cannot determine this. (Indicates a) j =t j -t j-1 For j≥1, the estimated rate of change can be solved as follows:

[0050]

[0051] Equation 1 is the maximum likelihood estimate of the rate of change of a given content item.

[0052] The second scenario occurs when complete change observations are available. As mentioned, change observations can be exported from the browser that accesses the content item and sends back the content information. In this case, for all j, z j=1, meaning that a change only occurs when a change is observed, and this observation precisely indicates a change. In this case, the rate of change can be estimated only from t0 to t... U The average number of changes observed during the observation period.

[0053] Once the rates of change are estimated, they can be used to generate new refresh schedules. These schedules can be followed, and additional change observations collected by following them can be added to the dataset. Updated rates of change can then be estimated.

[0054] The importance score calculation component 336 provides importance scores to other components of the system. The importance score assigned to a content item can take many forms and can be constructed using various characteristics as input. For example, spam score can be input for importance score calculation. The web business of a content item can be used as input for importance score; sites with high business are more important. The click-through rate of search results can be another factor; a high click-through rate leads to a higher importance score. Similarly, a higher frequency of queries related to a given content item may result in a higher importance score. In one aspect, various content items have already been assigned importance scores by search engines or other trackers (even if not called importance scores), and these scores can be used by various aspects of the technology as importance scores or as components of new scores. In one approach, different factors are given different weights to produce importance scores. In one aspect, importance scores are greater than zero because using zero within the cost function would result in zero cost for not accessing the content item. Zero cost could lead to zero-schedule crawling of the item, which is inconsistent with the goals of the technology described herein. If a content item should not be refreshed, it is best to exclude it from the content set being evaluated.

[0055] The incomplete change data scheduler 338 constructs a refresh schedule across content items with incomplete change data. The incomplete change data scheduler 338 can construct the refresh schedule by following the method 400 described below. Once constructed, the refresh schedule can be output to the crawler 344 for execution. In other words, the crawler 344 will use the refresh schedule constructed by the incomplete change data scheduler 338 to refresh the content within a time frame.

[0056] If a content item is notified of complete change data almost instantly whenever it changes, then that item is said to have complete change data; otherwise, it is said to have incomplete change data. Change data can be obtained by leveraging signals from users accessing different content. For example, browsers can provide webpage information, such as page hashes, to change analysis components. Frequently visited websites may generate substantially complete change data from the browsers of many visitors. Thresholds can be used to determine what level of change information is needed to classify an item as having complete change information. Thresholds can take into account both analysis frequency and historical change frequency. Any content that does not meet the complete change threshold is classified as incomplete.

[0057] The complete change data scheduler 340 constructs a refresh schedule across content items that have complete change data. The complete change data scheduler 340 can construct the refresh schedule following the method 500 described below. Once constructed, the refresh schedule can be output to the crawler 344 for execution. In other words, the crawler 344 will use the refresh schedule constructed by the complete change data scheduler 340 to refresh the content within the planned scope. The criteria used to determine whether a content item has complete change data have been previously described.

[0058] Hybrid scheduler 342 constructs refresh schedules across content items with both complete and incomplete change data. Hybrid scheduler 342 can construct refresh schedules following method 600. Once constructed, the refresh schedules can be output to crawler 344 for execution. In other words, crawler 344 will use the refresh schedules constructed by hybrid scheduler 342 to refresh content within a time frame. The criteria used to determine whether a content item has complete or incomplete change data have been previously described.

[0059] Crawler 344 comprises one or more web crawlers responsible for keeping the collection of content items fresh. Crawler 344 is typically associated with trackers such as search engines. Crawler 344 receives refresh schedules and follows them. Crawler 344 may collect additional data describing its refresh efforts. For example, crawler 344 may record each content item accessed, the access time, and whether the content item has changed at the time of access. This information can be used to provide an updated estimate of the rate of change.

[0060] Content cache 346 stores copies of content items retrieved by crawler 344. These copies may not be accurate. For example, metadata or other information from the content may be optionally omitted from the cache. In the case of search results, cache 346 can help search engines quickly retrieve content items or information derived from them.

[0061] Now go to Figure 4Based on various aspects of the technology described herein, a flowchart is provided illustrating a method 400 for preparing a schedule for a crawler to follow when seeking updated copies of content items in a content set. As previously described, method 400 can be performed by a search server or other computer component. Method 400 is suitable for preparing a schedule to crawl content items with incomplete change information.

[0062] In step 410, a bandwidth constraint is received, which limits the total amount of crawling a crawler is allowed across a content set within a fixed time unit (e.g., an hour or a day). A bandwidth constraint can have two components. The first component is the crawler constraint. The second component is the host constraint, sometimes called a courtesy constraint. A crawler constraint can be thought of as a bottleneck resulting from the combination of hardware and software involved in the refresh process. Specific bottlenecks may be limited to network bandwidth, processor bandwidth, memory bandwidth, software constraints, or some other system limitation that allows for so many crawling events per unit of time. A crawler constraint can also be a virtual constraint. For example, even if more network resources are theoretically available, in practice a subset of these available resources are allocated to a specific crawler. For example, different crawlers associated with a tracker may be responsible for refreshing different content sets. These crawlers may be allocated a specific amount of resources for these functions to be performed by system administrators, etc. The goal of refresh scheduling is to maximize the freshness of the content in the presence of any bandwidth constraints. In some aspects, crawler constraints are only considered when constructing an optimized refresh schedule.

[0063] The second constraint is imposed by the content host. Some hosts may limit the number of times a crawler can access a content host within a fixed time unit (e.g., an hour or a day). This constraint can apply to a set of content items that share a common content host. In other words, the limitation may not be per item, but rather across the set of items. A refresh schedule that does not consider host-imposed constraints may not utilize all available bandwidth because some crawling work will target content items that will eventually become inaccessible. Furthermore, content items may become stale in the long term if available crawling events are used up early in the planned timeframe. By distributing these limited crawling events more evenly, overall freshness can be maximized.

[0064] In step 420, an importance score set is received, which includes a separate importance score for each content item in the content set. The importance score assigned to a content item can take many forms and can be constructed using various characteristics as input. For example, spam score can be an input for calculating the importance score. The web business of a content item can be an input for the importance score; sites with high business are more important. The click-through rate of search results can be another factor; a high click-through rate leads to a higher importance score. Similarly, a higher frequency of queries associated with a given content item may result in a higher importance score. In one aspect, various content items have already been assigned importance scores by search engines or other trackers (even if not called importance scores), and these scores can be used by various aspects of the technology as importance scores or as components of new scores. In one approach, different factors are given different weights to produce importance scores. In one aspect, the importance score is greater than zero because using zero within the cost function would result in zero cost for not accessing the content item. Zero cost could lead to zero-schedule crawling of the item, which is inconsistent with the goals of the technology described herein. If a content item should not be refreshed, it is best to exclude it from the content set being evaluated.

[0065] In step 430, a set of rates of change is received, which includes the individual rate of change for each content item in the content set. The rate of change is estimated based on previous observations of the content items. Different estimators can be used, such as those previously referenced. Figure 3 The description generally assumes that future rates of change will resemble previously observed rates of change. Change observations can be obtained from crawl events and change telemetry. When a crawl event occurs, the crawler determines whether the content has changed since the last crawl event. Both crawl events are time-related. These times, indicating whether a change has occurred, can be used to estimate the rate of change. Various aspects of this technique can be used to calculate the estimated rate of change by assuming that the change is uniformly distributed over time. The rate of change can also be based on change telemetry received from the user or other sources. This change telemetry can be evaluated by the change notification component 332 to determine whether a meaningful change has occurred. Meaningful changes can be used to calculate the estimated rate of change. For example, the rate of change can be generated by the change frequency determiner 334 described earlier.

[0066] In step 440, the crawl rate scheduling of the content set is calculated using a cost function that includes the individual crawl rate for each individual content item in the content set. The set of rates of change and the set of importance scores are the inputs to the cost function. The crawl rate scheduling is a solution to the cost function that minimizes the sum of costs across all content items and is constrained by a bandwidth constraint such that the sum of the crawl rates of all content items in the content set equals the bandwidth constraint.

[0067] The cost function can be expressed as:

[0068]

[0069] J π It is the cost penalty of refreshing strategy π;

[0070] Among them W - It is a content set that includes content items with incomplete change information, and w is a single content item in the content set;

[0071] ρ w = Crawling rate of individual content items;

[0072] Δ w = Estimated rate of change for individual content items; and

[0073] μ w = Importance score of individual content items.

[0074] In one aspect, the Lagrange multiplier method is used to solve the cost function in Equation 2. In this implementation, the optimal refresh plan can be expressed as the following system of equations.

[0075] For all w∈W - ,

[0076] Where R is the bandwidth constraint; and

[0077] λ is a Lagrange multiplier.

[0078] The Lagrange multipliers can be computed using a split-search method or some other mechanism. In one aspect, a test Lagrange multiplier is selected, and the system of equations is solved. If the sum of the crawl rates of all items in the set exceeds the bandwidth constraint, the process is repeated using a higher test Lagrange multiplier. Otherwise, if the sum of the crawl rates of all items in the set does not meet the bandwidth constraint, the process is repeated using a lower test Lagrange multiplier. In this way, the optimal Lagrange multiplier is determined iteratively.

[0079] Furthermore, assuming that the rate of change and importance score are not zero, some costs of not accessing content items will be calculated using the equations shown.

[0080] In step 450, a crawl rate schedule is output for the crawler to use. The crawl rate schedule can be in the form of a crawl rate vector, where the crawl rate is specified for each content item in the content set. The crawler can then be used to execute the crawl rate schedule and retrieve fresh content. Fresh content can be added to the cache. Data collected during the execution of the crawl rate schedule can be gathered and used to improve the change rate of different items. A new crawl rate schedule can then be generated, further optimizing the crawl schedule.

[0081] Now go to Figure 5 Based on various aspects of the technology described herein, a flowchart is provided illustrating a method 500 for preparing a schedule for a crawler to follow when seeking updated copies of content items in a content set. As previously described, method 500 can be performed by search server 206 or other computer components. Method 500 is adapted to prepare a schedule for crawling content items with complete change information.

[0082] In step 510, a bandwidth constraint is received, which limits the total amount of crawling the crawler is allowed to perform across the content set per unit time. The bandwidth constraint can have two components. The first component is the crawler constraint. The second component is the host constraint, sometimes called a courtesy constraint. For example, these bandwidth constraints have been previously referenced. Figure 4 describe.

[0083] In step 520, an importance score set is received, which includes a separate importance score for each content item in the content set. The importance scores assigned to content items can take many forms and can be constructed using various characteristics as input. For example, the importance scores have previously been referenced... Figure 4 describe.

[0084] In step 530, a set of rates of change is received, which includes the individual rate of change for each content item in the content set. The rate of change is estimated based on previous observations of the content items. Different estimators can be used, such as those previously referenced. Figure 3 The description generally assumes that future rates of change will resemble previously observed rates of change. Change observations can be derived from crawling events and change telemetry. For example, the rate of change has previously been referenced... Figure 4 describe.

[0085] In step 540, the crawl probability vector is determined using bandwidth constraints, an importance score set, and a change rate set. This crawl probability vector includes an individual crawl probability for each individual content item in the content set. The individual crawl probability is used to determine whether to crawl the associated individual content item upon receiving a change notification. In other words, the system may not crawl every content item every time a change notification is received. Instead, the crawl probability is used to determine whether to access the site, with different probability values ​​translating into different values ​​for the cost function. Conceptually, this can be imagined as flipping a coin when a change notification is received to determine whether to access the site. Instead of the 50% probability associated with a coin toss, the crawl probability calculated for the item will be used to make the decision to crawl or not crawl. For example, a random number generator can be used to generate a number between 1 and 100 each time a change notification for a content item is received. A crawl probability of 0.6 can be achieved by crawling the content item when the randomly generated number is less than or equal to 60.

[0086] In one aspect, the cost function used to determine the crawling probability is

[0087]

[0088] Where W° is a content set including content items with complete change information, w is a single content item in the content set, and p w It is the probability that project w will refresh (crawl) when it receives a change notification, and u w It is its importance score.

[0089] Under any refresh strategy, the crawling rate of any content item is related to its rate of change. Each time a changed observation is received, crawling is initiated with a crawling probability p. w This occurs. Therefore, the crawling rate of a content item is the product of its crawling probability and its rate of change. The probability is expressed as a number between 0 and 1. Furthermore, bandwidth greater than the overall rate of change of all items in the content set does not provide any benefit, as this would result in crawling more items than changing them. Considering these inputs, the following system can be used to calculate the optimal freshness schedule:

[0090]

[0091] Subject to

[0092] For all w∈W°, ∑ w∈W° ρ w Δ w =R, 0≤1 (6)

[0093] Solving Equation 5 requires nonlinear optimization under inequality constraints, which can typically take time exponentially with the number of constraints. If the inequality constraints are ignored, the solution can be expressed as:

[0094] For all w∈W°,

[0095] In this example, This represents the crawling probability of a content item. Equation 6 indicates that the probability of a relaxed solution will never be less than 1, but it may indeed violate the constraint by having a probability greater than 1. The main difficulty in optimizing inequality constraints is determining the subset of inequality constraints that are active under the optimal solution, i.e., finding all content items w with an optimal probability of 1 in this case. These content items can be found iteratively. Whenever a single content item is calculated to have a probability greater than 1 via Equation 7, it is assigned a probability of 1 and set aside. The available bandwidth is then reduced by the estimated rate of change of the items, and this process is repeated for the remaining items. In this way, all content items are assigned a value greater than zero but less than 1.

[0096] In step 550, a crawl probability vector is output for use by the crawler, comprising an individual crawl probability for each content item. The probability vector can take the form of a crawl probability vector where the crawl probability is specified for each content item in the content set. Then, when a change notification arrives, the crawler can be used to execute the crawl probability vector by crawling content items with the corresponding probabilities, thereby retrieving fresh content. The fresh content can be added to the cache. Data collected during crawl rate scheduling can be gathered and used to improve the change rate of different items. A new crawl probability vector can then be generated to further optimize the crawl schedule.

[0097] Now go to Figure 6 Based on various aspects of the techniques described herein, a flowchart is provided illustrating a method 600 for preparing a schedule for a crawler to follow when seeking updated copies of content items in a content set. Preparing the schedule can include determining the optimal split of available crawling bandwidth between a first set of content items with incomplete changed data and a second set of content items with complete changed data.

[0098] In step 610, a bandwidth constraint is received, limiting the total amount of crawling the crawler is allowed to perform across a content set per unit of time. This content set comprises a first subset of content items with incomplete change information and a second subset of content items with complete change information. A content item has complete change data when a near-instantaneous change notification is received each time it changes; otherwise, it has incomplete change data. Change data can be obtained by leveraging signals from users accessing different content. For example, browsers can provide webpage information, such as page hashes, to the change analysis component. Frequently visited websites may have substantially complete change data provided by the browsers of many visitors. A threshold can be used to determine what level of change information is required to classify an item as having complete change information. The threshold can take into account both analysis frequency and historical change frequency. Any content that does not meet the complete change threshold is classified as incomplete.

[0099] Bandwidth constraints can have two components. The first component is the crawler constraint. The second component is the host constraint, sometimes called a courtesy constraint. For example, these bandwidth constraints have been previously referenced. Figure 4 describe.

[0100] In step 620, an importance score set is received, which includes a separate importance score for each content item in the content set. The importance scores assigned to content items can take many forms and can be constructed using various characteristics as input. For example, the importance scores have previously been referenced... Figure 4 describe.

[0101] In step 630, a set of rates of change is received, which includes the individual rate of change for each content item in the content set. The rate of change is estimated based on previous observations of the content items. Different estimators can be used, but it is generally assumed that future rates of change will resemble previously observed rates of change. Change observations can be derived from crawling events and change telemetry. For example, the rate of change has previously been referenced... Figure 4 describe.

[0102] In step 640, an optimized split of the available crawling bandwidth between the first and second subsets is determined. The available crawling bandwidth can be defined by the bandwidth constraints of step 610. The final result of the optimized split can be a first bandwidth allocation for the first subset and a second bandwidth allocation for the second subset. The optimized split can be determined by minimizing a combined cost function, which includes a first cost function for the first subset and a second cost function for the second subset. The first and second cost functions are different from each other. This requires combining the first and second cost functions into a single cost function that can be solved together, as shown:

[0103]

[0104] This can be reiterated as follows:

[0105]

[0106] Subject to

[0107] For all W∈W°, ∑ w∈W° ρ w Δ w =R, 0≤1 (10)

[0108] and

[0109] For all w∈W - p w >0, for all w∈W°, 0 <p w ≤1 (11)

[0110] The optimization objective (Equation 9) is strictly concave, being the sum of concave functions over the region described by the constraints, and therefore possesses a unique maximizer. To efficiently find the maximizer, we observe that solving Equation 9 is equivalent to deciding how to split the global bandwidth constraint R into an allocation R° of content with fully varied observations and Rˉ=RR° of the remaining content. By iteratively guessing the splits, evaluating the splits by adding the costs calculated by the first and second cost functions, and generating a next guessed split that improves upon the previous split, the optimal split can be found numerically efficiently.

[0111] The iterative method involves repeating the following steps until an optimal solution is found. The first step is a test split of the received or generated bandwidth, including a first test bandwidth for a first content set and a second test bandwidth for a second content set. Each iteration will have a different test split. Next, the first optimization schedule for the first subset of content items uses the first test bandwidth, the importance score from step 620, and the rate of change from step 630 as inputs. Figure 4 The method described is used for calculation. Figure 5 The method can be used to compute the first optimal probability vector using the second test bandwidth, the importance score from step 620, and the rate of change from step 630 as inputs. A first cost function is then used to compute the total cost of the first optimized test schedule. A second cost function is then used to compute the total cost of the first optimal probability vector. These two costs are summed to generate the total cost of the first iteration. These steps are repeated for additional iterations until the minimum total cost is found. The split associated with the iteration having the lowest total cost is the optimal split.

[0112] In step 650, a first bandwidth allocation for calculating the crawl rate scheduling of the first subset and a second bandwidth allocation for calculating the crawl probability vector of the second subset are output.

[0113] Once the optimal split is determined, method 400 can be used to calculate the optimal crawl rate schedule for content items with incomplete change data, and method 500 can be used to determine the probability vector for content items with complete change data. The crawl rate schedule and crawl probability vector are output for the crawler to use. The crawl rate schedule can be in the form of a crawl rate vector, where the crawl rate is specified for each content item in the content set. The crawler can then be used to execute the crawl rate schedule and retrieve fresh content. Fresh content can be added to the cache. Data collected during the execution of the crawl rate schedule can be gathered and used to improve the change rate of different items. A new crawl rate schedule can then be generated, further optimizing the crawl schedule.

[0114] The probability vector can take the form of a crawl probability vector, where the crawl probability is specified for each content item in the content set. The crawler can then be used to execute the crawl probability vector and retrieve fresh content. Fresh content can be added to the cache. Data collected during crawl rate scheduling can be gathered and used to improve the rate of change for different items. New crawl probability vectors can then be generated, further optimizing the crawl schedule.

[0115] Exemplary computing environment

[0116] For reference Figure 3 The techniques described herein can be implemented in a distributed manner, enabling them to run on computer clusters / networks that work together to perform... Figures 1 to 6 The calculations described. In one aspect, Figure 7 The described computing device 700 can be one of multiple computers in a cluster. (See reference...) Figure 7 The computing device 700 includes a bus 710 that directly or indirectly couples to the following devices: memory 712, one or more processors 714, one or more presentation components 716, one or more input / output (I / O) ports 718, one or more I / O components 720, and an illustrative power supply 722. Bus 710 can represent one or more buses (such as an address bus, a data bus, or a combination thereof). Although Figure 7 The various boxes are shown with lines for clarity, but in reality, these boxes represent logical components, not necessarily actual components. For example, a presentation component (such as a display device) can be considered an I / O component. Furthermore, the processor has memory. The inventors hereby recognize this as the nature of the art and reiterate... Figure 7The accompanying drawings illustrate only exemplary computing devices that can be used in conjunction with one or more aspects of this technology. There is no distinction between categories such as "workstation," "server," "laptop," and "handheld device," as all of these are envisioned in... Figure 7 Within the scope and refer to "Computing Devices".

[0117] Computing device 700 typically includes a variety of computer-readable media. Computer-readable media can be any available medium accessible by computing device 700, and includes volatile and non-volatile media, removable and non-removable media. By way of example and not limitation, computer-readable media can include computer storage media and communication media.

[0118] Computer storage media includes volatile and non-volatile, removable and non-removable media implemented using any method or technology for storing information (such as computer-readable instructions, data structures, program modules or other data). Computer storage media includes, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, Digital Universal Disc (DVD) or other optical disc storage devices, magnetic cartridges, magnetic tapes, disk storage devices or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by the computing device 700. Computer storage media itself does not include signals.

[0119] Communication media typically implement computer-readable instructions, data structures, program modules, or other data by modulating data signals using mechanisms such as carrier waves or other transmission mechanisms, and include any information delivery medium. The term "modulated data signal" refers to a signal whose one or more characteristics are set or altered in a manner that encodes information in the signal. By way of example, and not limitation, communication media include wired media (such as wired networks or direct wired connections) and wireless media (such as acoustic, RF, infrared, and other wireless media). Any combination of these should also be included within the scope of computer-readable media.

[0120] Memory 712 includes computer storage media in the form of volatile and / or non-volatile memory. Memory can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. Computing device 700 includes one or more processors 714 that read data from various entities, such as memory 712 or I / O components 720. Multiple presentation components 716 present data instructions to a user or other device. Exemplary presentation components include display devices, speakers, printing components, vibration components, etc.

[0121] I / O port 718 allows computing device 700 to be logically coupled to other devices including I / O components 720, some of which may be built-in. Illustrative components include microphones, joysticks, gamepads, satellite dishes, scanners, printers, wireless devices, etc.

[0122] I / O component 720 can provide a Natural User Interface (NUI) that processes user-generated air gestures, voice, or other physiological input. In some instances, the input can be transmitted to appropriate network elements for further processing. The NUI can implement any combination of voice recognition, touch and stylus recognition, facial recognition, biometric recognition, on-screen and near-screen gesture recognition, air gestures associated with a display on computing device 700, head and eye tracking, and touch recognition. Computing device 700 can be equipped with a depth camera, such as a stereo camera system, an infrared camera system, an RGB camera system, and combinations thereof, for gesture detection and recognition. Additionally, computing device 700 can be equipped with an accelerometer or gyroscope capable of motion detection. The output of the accelerometer or gyroscope can be provided to the display of computing device 700 to render immersive augmented reality or virtual reality.

[0123] Some aspects of computing device 700 may include one or more radios 724 (or similar wireless communication components). Radios 724 transmit and receive radio or wireless communications. Computing device 700 may be a wireless terminal suitable for receiving communications and media via various wireless networks. Computing device 700 may communicate with other devices via wireless protocols such as Code Division Multiple Access (“CDMA”), Global System for Mobile Communications (“GSM”), or Time Division Multiple Access (“TDMA”) and other wireless protocols. Radio communication may be a short-range connection, a long-range connection, or a combination of short-range and long-range wireless telecommunication connections. When we refer to “short” and “long” types of connections, we are not referring to a spatial relationship between two devices. Rather, we generally refer to short-range and long-range as different categories or types of connections (i.e., primary connections and secondary connections). By way of example and not limitation, a short-range connection may include communication with a device that provides access to a wireless communication network (e.g., a mobile hotspot). Connections, such as WLAN connections using the 802.11 protocol; and Bluetooth connections to another computing device are further examples of short-range or near-field communication connections. By way of example and not limitation, long-range connections may include connections using one or more of the CDMA, GPRS, GSM, TDMA, and 802.16 protocols.

[0124] Now go to Figure 8Based on various aspects of the techniques described herein, a flowchart is provided illustrating a method 800 for preparing probability vectors for a crawler to follow when seeking updated copies of content items in a content set. As previously described, method 800 can be performed by search server 206 or other computer components. Method 800 is adapted to prepare probability vectors for content items with complete change information.

[0125] In step 810, a crawl probability vector is determined, which includes an individual crawl probability for each individual content item in the content set. The individual crawl probability is used to determine whether to crawl the associated individual content item upon receiving a change notification, using bandwidth constraints, a set of importance scores, and a set of change rates. In other words, the system may not crawl every content item every time a change notification is received. Instead, the crawl probability is used to determine whether to access the site, with different crawl probability values ​​translating into different values ​​for the cost function. Conceptually, this can be imagined as flipping a coin when a change notification is received to determine whether to access the site. Instead of the 50% probability associated with a coin toss, the crawl probability calculated for the item will be used to make the decision to crawl or not crawl. For example, a random number generator can be used to generate a number between 1 and 100 each time a change notification for a content item is received. A crawl probability of 0.6 can be achieved by crawling the content item when the randomly generated number is less than or equal to 60.

[0126] Different inputs can be used to determine the crawling probability. For example, a bandwidth constraint can be received, which limits the total amount of content a crawler is allowed to crawl per unit of time. A bandwidth constraint can have two components. The first component is the crawler constraint. The second component is the host constraint, sometimes called a courtesy constraint. These bandwidth constraints have been previously referenced, for example. Figure 4 describe.

[0127] A set of importance scores can be received, comprising a separate importance score for each content item in the content set. The importance scores assigned to content items can take many forms and can be constructed using various characteristics as input. For example, importance scores have previously referenced... Figure 4 describe.

[0128] A set of rates of change is received, which includes the individual rate of change for each content item in the content set. The rates of change are estimated based on previous observations of the content items. Different estimators can be used, such as those from previous references. Figure 3 The description generally assumes that future rates of change will resemble previously observed rates of change. Change observations can be derived from crawling events and change telemetry. For example, the rate of change has previously been referenced... Figure 4 describe.

[0129] In one aspect, the cost function used to determine the crawling probability is

[0130]

[0131] Where W° is a content set including content items with complete change information, w is a single content item in the content set, and p w It is the probability that project w will refresh (crawl) when it receives a change notification, and mu w It is its importance score.

[0132] Under any refresh strategy, the crawling rate of any content item is related to its rate of change. Each time a changed observation is received, crawling is initiated with a crawling probability p. w This occurs. Therefore, the crawling rate of a content item is the product of its crawling probability and its rate of change. The probability is expressed as a number between 0 and 1. Furthermore, bandwidth greater than the overall rate of change of all items in the content set does not provide any benefit, as this would result in crawling more items than changing them. Considering these inputs, the following system can be used to calculate the optimal freshness schedule:

[0133]

[0134] Subject to

[0135] For all w∈W°, ∑ w∈W° ρ w Δ w =R, 0≤1 (6)

[0136] Solving Equation 5 requires nonlinear optimization under inequality constraints, which can typically take time exponentially with the number of constraints. If the inequality constraints are ignored, the solution can be expressed as:

[0137] For all w∈W°,

[0138] In this example, This represents the crawling probability of a content item. Equation 6 indicates that the probability of a relaxed solution will never be less than 1, but it may indeed violate the constraint by having a probability greater than 1. The main difficulty in optimizing inequality constraints is determining the subset of inequality constraints that are active under the optimal solution, i.e., finding all content items w with an optimal probability of 1 in this case. These content items can be found iteratively. Whenever a single content item is calculated to have a probability greater than 1 via Equation 7, it is assigned a probability of 1 and set aside. The available bandwidth is then reduced by the estimated rate of change of the items, and this process is repeated for the remaining items. In this way, all content items are assigned a value greater than zero but less than 1.

[0139] In step 820, a crawl probability vector is output for use by the crawler, which includes an individual crawl probability for each content item. The probability vector can take the form of a crawl probability vector where the crawl probability is specified for each content item in the content set. Then, when a change notification arrives, the crawler can be used to execute the crawl probability vector by crawling the content items with the corresponding probabilities, thereby retrieving fresh content. The fresh content can be added to the cache. Data collected during crawl rate scheduling can be gathered and used to improve the change rate of different items. A new crawl probability vector can then be generated to further optimize the crawl schedule.

[0140] Many different arrangements of the various components depicted, as well as components not shown, are possible without departing from the scope of the following claims. Embodiments of this disclosure have been described in an illustrative rather than restrictive manner. Alternative embodiments will become apparent to the reader of this disclosure upon reading it and as a result of reading it. Alternative means of achieving the foregoing can be accomplished without departing from the scope of the following claims. Certain features and sub-combinations are useful and can be used without reference to other features and sub-combinations, and are contemplated within the scope of the claims.

Claims

1. A method for updating copies of content items in a content set, the method comprising: receiving a bandwidth constraint that limits a total amount of crawling allowed across the content set per unit of time, wherein the content set includes a first subset of content items having incomplete change information and a second subset of content items having complete change information; receiving a set of importance scores that includes a separate importance score for each content item in the content set; receiving a set of change rates that includes a separate change rate for each content item in the content set; determining an optimal split of bandwidth between the first subset and the second subset by minimizing a cost function associated with the first subset and the second subset to produce a first bandwidth allocation for the first subset and a second bandwidth allocation for the second subset; outputting the first bandwidth allocation for use in computing a crawl rate schedule for the first subset and outputting the second bandwidth allocation for use in computing a crawl probability vector for the second subset; computing the crawl rate schedule for the first subset using a first cost function, the crawl rate schedule including a separate crawl rate for each separate content item in the first subset, wherein the set of change rates and the set of importance scores are inputs to the first cost function, wherein the crawl rate schedule is a solution to the first cost function that minimizes a sum of costs across all content items in the first subset and is constrained by the first bandwidth allocation such that a sum of crawl rates for all content items in the first subset equals the first bandwidth allocation; and retrieving updated copies of content items in the first subset according to the crawl rate schedule and retrieving updated copies of content items in the second subset according to the crawl probability vector.

2. The method of claim 1, wherein the bandwidth constraint includes a crawler constraint and a host constraint, wherein the crawler constraint is defined as a first sum of crawl events a crawler is allowed to perform per unit of time for all content sources, and wherein the host constraint is a second sum of crawl events allowed per unit of time for a separate host content source.

3. The method of claim 1, wherein the separate change rate is estimated based on an analysis of previous crawl events for the separate content item and a determination of whether the separate content item changed between crawl events.

4. The method of claim 3, wherein the first cost function is minimized using a Lagrange multiplier method.

5. The method of claim 4, wherein a Lagrange multiplier used in the Lagrange multiplier method is determined using a bisection search method.

6. The method of claim 1, wherein change information for content items in the content set is incomplete.

7. The method of claim 1, wherein the separate crawl rate for each separate content item in the content set is greater than zero. ​ 8. A method for seeking updated copies of content items in a content collection by following a crawl probability vector, the method comprising: receiving a bandwidth constraint that limits a total amount of crawling allowed across the content collection per unit of time, wherein the content collection includes a first subset of content items having incomplete change information and a second subset of content items having complete change information; receiving a set of importance scores that includes a separate importance score for each content item in the content collection; receiving a set of change rates that includes a separate change rate for each content item in the content collection; determining an optimal split of bandwidth between the first subset and the second subset by minimizing a cost function associated with the first subset and the second subset to produce a first bandwidth allocation for the first subset and a second bandwidth allocation for the second subset; outputting the first bandwidth allocation for use in computing a crawl rate schedule for the first subset and outputting the second bandwidth allocation for use in computing a crawl probability vector for the second subset; computing the crawl probability vector for the second subset using a second cost function, the crawl probability vector including a separate crawl rate for each separate content item in the second subset, wherein the set of change rates and the set of importance scores are inputs to the second cost function, wherein the crawl probability vector is a solution to the second cost function that minimizes a sum of costs across all content items in the second subset and is constrained by the second bandwidth allocation such that a sum of crawl rates for all content items in the second subset equals the second bandwidth allocation; and retrieving updated copies of content items in the first subset according to the crawl rate schedule and retrieving updated copies of content items in the second subset according to the crawl probability vector.

9. The method of claim 8, further comprising: computing the crawl rate schedule for the first subset using a first cost function, the crawl rate schedule including a separate crawl rate for each separate content item in the first subset, wherein the set of change rates and the set of importance scores are inputs to the first cost function, wherein the crawl rate schedule is a solution to the first cost function that minimizes a sum of costs across all content items in the first subset and is constrained by the first bandwidth allocation such that a sum of crawl rates for all content items in the first subset equals the first bandwidth allocation.

10. The method of claim 9, wherein the cost function is solved using a Lagrangian multiplier method.

11. The method of claim 8, wherein the bandwidth constraint includes a crawler constraint and a host constraint, wherein the crawler constraint is defined as a first sum of crawl events a crawler is allowed to perform per unit of time for all content sources, and wherein the host constraint is a second sum of crawl events allowed per unit of time for a separate host content source. ​ 12. The method of claim 8, wherein the individual change rate is estimated based on an analysis of previous crawl events for the individual content item and a determination of whether the individual content item changed between crawl events.

13. One or more computer storage media that, when executed by a computing device, cause the computing device to perform a method for seeking updated copies of content items in a set of content, the method comprising: receiving a bandwidth constraint that limits a total amount of crawling that a crawler is allowed to perform across the set of content per unit of time, wherein the set of content includes a first subset of content items having incomplete change information and a second subset of content items having complete change information; receiving a set of importance scores that includes an individual importance score for each content item in the set of content; receiving a set of change rates that includes an individual change rate for each content item in the set of content; determining an optimal split of bandwidth between the first subset and the second subset by minimizing a cost function associated with the first subset and the second subset to produce a first bandwidth allocation for the first subset and a second bandwidth allocation for the second subset; outputting the first bandwidth allocation for use in computing a crawl rate schedule for the first subset and outputting the second bandwidth allocation for use in computing a crawl probability vector for the second subset; and retrieving updated copies of content items in the first subset according to the crawl rate schedule and retrieving updated copies of content items in the second subset according to the crawl probability vector.

14. The media of claim 13, further comprising: using the bandwidth constraint, the set of importance scores, and the set of change rates, computing the crawl probability vector using a second cost function for the second subset, the crawl probability vector including an individual crawl probability for each individual content item in the second subset, wherein the individual crawl probability is used to determine whether to crawl an associated individual content item upon receiving a change notification for the associated individual content item; wherein the crawl probability vector is a solution to the second cost function that minimizes a second sum of costs across all content items in the second subset and is constrained by the bandwidth constraint such that a second sum of crawl rates for all content items in the second subset equals the bandwidth constraint minus bandwidth allocated for crawling the first subset; and outputting the crawl probability vector.

15. The media of claim 13, further comprising: computing the crawl rate schedule for the first subset using a first cost function for the first subset, the crawl rate schedule including a separate crawl rate for each separate content item in the first subset of content items, wherein the set of change rates and the set of importance scores are inputs to the first cost function, wherein the crawl rate schedule is a solution to the first cost function: the solution minimizes a sum of costs across all content items and is constrained by the bandwidth constraint, such that a sum of crawl rates for all content items in the first subset equals the first bandwidth allocation; and outputting the crawl rate schedule.

16. The medium of claim 14, wherein the separate change rate is estimated based on an analysis of previous crawl events for the separate content item and a determination of whether the separate content item changed between crawl events.

17. The medium of claim 13, wherein first change information for a first subset of content items in the set of content items is incomplete and second change information for a second subset of content items in the set of content items is complete.

Images