Distributed systems for improving task-machine cohesion

Abstract

A technique for allocating tasks within a distributed system is described herein. The distributed system includes a plurality of computing devices. Each computing device in turn provides a bucket manager and at least one application instance. The bucket manager allocates work to the computing devices in units of buckets in a manner managed by a set of application-specific initialization parameters. The bucket managers also collaborate with a separate storage and notification (S&N) system to perform their allocation functions. The S&N system stores data nodes in a hierarchical data structure that reflects the allocation-related state of the distributed system. The S&N system also notifies the bucket manager when the data nodes change. According to one provision, the bucket manager includes logic circuitry that reduces expensive context changes when computing devices join and leave the distributed system.

Description

Background Art

[0001] Distributed systems employ multiple computing devices to process data. These systems can use different strategies to assign tasks to computing devices. In the traditional master-slave model, distributed systems use one or more dedicated master computing devices to assign tasks to slave computing devices. In a decentralized approach, distributed systems do not use dedicated master computing devices to perform task assignment. For example, a distributed system can use a combination of consistent hashing and gossip protocols to assign tasks to machines. However, these strategies have various inefficiencies. Summary of the invention

[0002] This article describes a technique for allocating tasks within a distributed system. The distributed system includes multiple computing devices. Each computing device in turn provides a bucket manager and at least one application instance. The bucket manager allocates work to the computing devices in units of buckets in a manner managed by a set of application-specific initialization parameters. The bucket managers also collaborate with separate storage and notification (S&N) systems to perform their allocation functions. The S&N system stores data nodes in a hierarchical data structure that reflects the allocation-related state of the distributed system. The S&N system also notifies the bucket manager when a data node changes.

[0003] More specifically, the bucket manager can operate in a leader role and a follower role. When operating in leader mode, the bucket manager performs the following operations: interacts with the S&N system to determine whether a membership change has occurred in the distributed system; ensures that any detected change is stable; after confirming that the change is stable, updates the allocation of buckets to computing devices in the distributed system so that no two computing devices are assigned the same bucket; interacts with the S&N system to record the allocation of buckets. Each bucket corresponds to a quota of work to be performed by the computing devices assigned to it.

[0004] When operating in a follower role, the bucket manager of the computing device performs the following operations: interacts with the S&N system to determine whether there is a change in the buckets assigned to the computing device; notifies the application instance provided by the computing device of any detected changes to the buckets; and interacts with the S&N system to claim ownership of any buckets that have been newly assigned, and / or relinquish ownership of any buckets that have been newly removed. When an application instance receives notification that a new bucket has been assigned to it, it loads all resources required to perform tasks associated with the bucket. When an application instance receives notification that an existing bucket has been removed, it removes those resources.

[0005] In one implementation, the work routing component routes the task to the specific bucket by hashing the data associated with the task and then mapping the hash to a specific bucket ID. The work routing component then sends the task request to the computing device that owns the bucket associated with the specific bucket ID. More generally, each bucket handles all tasks that have a hash mapped to that bucket.

[0006] According to one technical advantage, the bucket manager corresponds to a generic resource that different applications can configure to allocate buckets to computing devices via corresponding initialization parameter sets. The use of a generic bucket manager reduces the burden on application developers who would otherwise need to incorporate custom allocation logic into their applications.

[0007] According to another technical advantage, the bucket manager can use timeout periods and other provisions to ensure that the computing device is stable before assigning a task to the computing device. This behavior promotes the tightness or "stickiness" of the task to the machine. This provision is useful because it reduces the amount of work (and the time associated with it) required to load the resources required to perform the task (for the newly joined computing device) and remove these resources (for the newly detached computing device).

[0008] According to another technical advantage, a bucket manager can perform its allocation function without interaction between computing devices in a distributed system. Instead of device-to-device interaction, the distributed system achieves this goal by using an S&N system to coordinate its allocation operations. This provision reduces the chance that any computing device will become overloaded due to the work performed by its bucket manager.

[0009] The techniques summarized above may be manifested in various types of systems, apparatus, components, methods, computer-readable storage media, data structures, graphical user interface presentations, articles of manufacture, and the like.

[0010] This Summary is provided to introduce some concepts in a simplified form; these concepts are further described below in the Detailed Description. This Summary is not intended to identify key features or essential features of the claimed subject matter, nor is it intended to be used to limit the scope of the claimed subject matter. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 An illustrative organization of logical components provided by a distributed system is shown.

[0012] Figure 2 An illustrative hierarchical organization of data nodes provided by a storage and notification (S&N) system is shown. The S&N system corresponds to Figure 1 One of the logical components.

[0013] Figure 3 A technique for allocating a leader computing device is shown. The technique utilizes Figure 2 Information provided by S&N Systems.

[0014] Figure 4 Shows the use of Figure 1 An illustrative distributed system of logical components.

[0015] Figures 5 to 7 shows the method for routing tasks to Figure 4 Figure 1 shows three illustrative working routing components for buckets in a distributed system.

[0016] Figure 8 Shows the use of Figure 1 The logical component of the upload service system.

[0017] Fig. 9 Operations performed by a bucket manager when a computing device boots up are shown.

[0018] Figures 10 to 12 The operations performed by a bucket manager when it assumes the role of leader are shown.

[0019] Fig.13 and Fig.14 The operations performed by a bucket manager when it plays the role of a follower are shown.

[0020] Fig.15 and Fig.16 Shows that when a notification of a new bucket is received (see Fig.15 ) and when receiving a notification to remove an existing bucket (see Fig.16 )An operation performed by an application instance.

[0021] Fig.17 and 18 Provides an overview of the operations that a bucket manager performs in its leader and follower roles.

[0022] Fig.19 Illustrative types of computing devices are shown that can be used to implement any aspect of the features shown in the preceding figures.

[0023] The same reference numerals are used throughout the disclosure and drawings to refer to the same parts and features. Figure 1 The series 200 numbers refer to features originally found in Figure 2 Features found in the series 300 numbers refer to the features originally found in Figure 3 Features found in , and so on. DETAILED DESCRIPTION

[0024] This disclosure is organized as follows. Section A describes a distributed system that uses a bucket manager to distribute work to computing devices. Section B sets forth an exemplary method for explaining the operation of the distributed system of Section A. Section C describes an illustrative computing device that can be used to implement any aspect of the features described in Sections A and B.

[0025] As a preliminary matter, the term "hardware logic circuit" includes one or more hardware processors (e.g., CPU, GPU, etc.) that execute machine-readable instructions stored in memory, and / or one or more other hardware logic units (e.g., FPGA) that perform operations using a task-specific collection of fixed and / or programmable logic gates. Section C provides additional information about one implementation of the hardware logic circuit. In some contexts, each of the terms "component", "tool", and "engine" refers to a portion of the hardware logic circuit that performs a specific function.

[0026] In one case, the separation of various components into different units shown in the figures can reflect the use of corresponding different physical and tangible components in actual implementation. Alternatively or additionally, any single component shown in the drawings can be implemented by multiple actual physical components. Alternatively or additionally, the depiction of any two or more separate components in the figures can reflect different functions performed by a single actual physical component.

[0027] Other figures describe concepts in the form of flow charts. In this form, certain operations are described as constituting different blocks that are executed in a specific order. Such an implementation is illustrative and not restrictive. Certain blocks described herein may be grouped together and executed in a single operation, certain blocks may be divided into a plurality of constituent blocks, and certain blocks may be executed in an order different from the order shown herein (including the manner in which these blocks are executed in parallel). In one implementation, the blocks related to the processing-related functions shown in the flow chart may be implemented by the hardware logic circuits described in Part C, which in turn may be implemented by one or more hardware processors and / or other logic units including a task-specific set of logic gates.

[0028] As for terminology, the phrase "configured to" includes various physical and tangible mechanisms for performing the identified operations. These mechanisms can be configured to perform the operations using the hardware logic circuits of Part C. The term "logic" likewise includes various physical and tangible mechanisms for performing tasks. For example, each processing-related operation shown in the flowchart corresponds to a logic component for performing that operation. The logic component can use the hardware logic circuits of Part C to perform its operations. When the logic component is implemented by a computing device, the logic component represents an electrical element that is a physical part of the computing system in any manner implemented.

[0029] Any storage resource or any combination of storage resources described herein may be considered a computer-readable medium. In many cases, a computer-readable medium represents a physical and tangible entity of some form. The term computer-readable medium also includes, for example, propagation signals sent or received via physical conduits and / or air or other wireless media. However, the specific term "computer-readable storage medium" explicitly excludes propagation signals per se, and includes all other forms of computer-readable media.

[0030] The following description may identify one or more features as "optional." This type of statement should not be interpreted as an exhaustive indication of features that may be considered optional; that is, other features may be considered optional despite not being explicitly identified as such in the text. Furthermore, any description of a single entity is not intended to preclude the use of multiple such entities; similarly, a description of multiple entities is not intended to preclude the use of a single entity. Furthermore, while this specification may interpret certain features as alternative ways of performing an identified function or implementing an identified mechanism, these features may also be combined together in any combination. Finally, the terms "exemplary" or "illustrative" refer to one implementation among potentially many implementations.

[0031] A. Illustrative Distributed Systems

[0032] Figure 1 A hierarchical structure 102 of logical components (eg, a software stack) including a bucket manager 104, an application 106, and an S&N system 108 is shown. Figure 4 As described more fully, each computing device of the distributed system implements an instance of bucket manager 104 and an instance of application 106. One or more separate computing devices may implement S&N system 108.

[0033] Bucket manager 104 assigns buckets to computing devices in a distributed system. Buckets partition a set of tasks processed by a computing device to which the bucket is assigned. For example, a bucket may correspond to a segment of a hash space. A computing device that includes a particular bucket will process all tasks that have a hash in the hash space that maps to the associated portion of the bucket.

[0034] Application 106 uses the allocated buckets in the following manner. Figure 1 106 ). The work routing component then maps the hash to a bucket ID. The bucket ID in turn corresponds to a specific bucket. The work routing component routes the task request to the computing device that currently owns the specific bucket. The computing device then processes the task using its local instantiation of the application 106.

[0035] More specifically, consider the merely exemplary case where the task involves processing data associated with a website. The work routing component may generate a hash of the uniform resource locator (URL) of the website, and then extract the n most significant bits of the hash, such as by shifting the hash right a specified number of positions, to produce a bucket ID. The bucket ID in turn identifies a particular bucket. The work routing component then routes the task request associated with the website to the computing device that currently owns the identified bucket.

[0036] Bucket manager 104 may sometimes operate in a leader role. In this role, bucket manager 104 uses leader role component 110 to manage the allocation and removal of buckets between computing devices in the distributed system. At any given time, the distributed system only allocates one bucket manager operating in the leader role.

[0037] Each bucket manager also operates in follower mode. In this role, the bucket manager 104 uses the follower role component 112 to receive and act on the allocation instructions generated by the leader. More specifically, each bucket manager also operates in a follower role, even during those times when the bucket manager also acts as a leader.

[0038] Applications 106 may use application logic 114 to perform any function or combination of functions. For example, in the above example, application logic 114 performs the process of converting website data into a form for upload to a distributed indexing system. Application logic 114 performs these operations on a set of buckets 116 having one or more buckets assigned to it at any given time. In some examples, application logic 114 also uses resources provided in data repository 118 to perform its operations. Resources may include data, program code, etc. For example, assume that application logic 114 performs some functions that rely on map information associated with the user's current location; in this case, the resources may correspond to a relatively large map file.

[0039] After the bucket manager 104 indicates ownership of a new bucket, the application logic 114 must perform the potentially time-consuming task of loading the appropriate application resources into the data store 118. When the bucket manager 104 indicates removal of a bucket, the application logic 114 must perform the potentially time-consuming task of clearing application resources from the data store 118. These events are more generally referred to herein as context changes.

[0040] The S&N system 108 includes a storage and retrieval component 120 and a notification component 122. The storage and retrieval component 120 stores and removes data nodes in a data repository 124 in accordance with instructions from the bucket manager 104. As will be described in more detail below, a data node generally describes a distribution-related state of a distributed system. The notification component 122 notifies the bucket manager 104 of any changes that have occurred to the data nodes in the data storage device 124. The notification component 122 performs this task in response to receiving a monitoring request (callback request) from the bucket manager 104. In one non-limiting implementation, the S&N system 108 can use the Zookeeper architecture described in "Zookeeper: Wait-free Coordination for internet-scale Systems" by Hunt et al., on page 14 of the proceedings of the 2010 USENIX Annual Technical Conference 2010. Note, however, that the Zookeeper architecture specifies a general framework for storing data in response to direct interaction with an application (which defines what data is stored), while Figure 1 The architecture of shows that application 106 interacts with S&N system 108 via the intermediary of bucket manager 104. Further note that the Zookeeper architecture simply stores data without knowing what the data represents; the application attaches meaning to the data. In other implementations, bucket manager 104 can work in conjunction with other S&N systems besides the S&N system that employs the Zookeeper architecture.

[0041] In one implementation, bucket manager 104 interacts with S&N system 108 through a set of application programming interfaces (APIs) provided by S&N system 108. Application 106 interacts with bucket manager 104 through a set of APIs provided by bucket manager 104.

[0042] According to one technical advantage, the hierarchical structure 102 of logical components uses a general bucket manager 104 to manage the allocation of buckets to applications 106, rather than relying on each application developer to create custom logic that performs this function at the application level. Therefore, this design simplifies the design of applications. Application developers only need to provide an interface that allows applications 106 to interact with bucket managers 104, such as through an API provided by bucket managers 104.

[0043] In practice, the application developer also generates a set of initialization parameters that will govern the operation of the bucket manager 104. When the bucket manager 104 is initialized, the application 106 can forward the set of initialization parameters to the bucket manager 104. As used herein, the term "parameter" refers to a pair consisting of a parameter variable (such as a timeout period) and a parameter value (such as 1 second).

[0044] In a non-limiting case, one such parameter specifies a timeout period that controls when the bucket manager 104 can consider a newly introduced computing device ready to accept a bucket. Another parameter specifies a timeout period that controls when the bucket manager 104 can consider a recently removed computing device to have truly left the distributed system. Another parameter specifies the optimal number of buckets that each computing device preferably has. Another parameter specifies the maximum number of buckets that each computing device will accept. Another parameter specifies the minimum number of buckets that each computing device will accept. Another parameter specifies whether a subset of computing devices can be operated as a backup computing device that does not accept a bucket until a specified failure condition is encountered. One or more other parameters may specify the amount of time that the distributed system is allowed to exceed one or more other thresholds (e.g., by exceeding the optimal number of buckets). Another parameter specifies the number of hash buckets to be used (e.g., 256 buckets, 512 buckets, etc.). Another parameter specifies the number of computing devices that the application plans to use, and so on. These parameters are described by way of example and not limitation; other implementations may use other parameters not mentioned above, and / or may omit one or more of the parameters mentioned above.

[0045] Note also that the application 106 does not interact directly with the S&N system 108. Instead, the application 106 interacts directly with the bucket manager 104, which in turn interacts with the S&N system 108. This provision also simplifies the task of application developers creating applications, for example, because the application developer does not need to build a custom interface for interacting with the S&N system 108.

[0046] According to another technical advantage, the computing devices that make up the distributed system do not need to communicate with each other in the process of managing buckets. Instead, the computing devices coordinate their behavior indirectly via the mediation of the S&N system 108. This arrangement reduces the load on any computing device, especially in the case where the computing device has a bucket manager that operates as a leader.

[0047] According to another technical advantage, the bucket manager 104 can be configured to reduce the number of context changes caused by adding and removing buckets. As will be explained more fully below, the bucket manager 104 performs this task in part by applying one or more timeout periods to ensure that the newly added computing device is stable before a bucket is assigned to it, and / or to ensure that the newly assigned device has actually left before its bucket is removed. Stated in another way, these provisions hinder unnecessary or inefficient changes in the context and have the effect of promoting closeness or "stickiness" between the tasks performed by the computing device and the computing device itself. This is particularly beneficial in those cases where the application 106 spends a lot of time loading and / or removing resources from the data repository 118. At the same time, the bucket manager 104 remains sufficiently responsive to changes in the distributed system to prevent computing devices that actually leave the distributed system from claiming ownership of the bucket, which may negatively affect performance.

[0048] Figure 1 The architecture shown is superior to the master-slave architecture for distributing tasks because the master-slave architecture requires the use of a dedicated master computing device. When the master computing device fails or becomes overloaded, the master-slave framework can provide subpar performance. Figure 1 The architecture shown has advantages over other distributed allocation algorithms because, as described above, it promotes closeness between tasks and computing devices and thus reduces the number of expensive context changes. It also provides a generic bucket manager that can be "plugged in" to different applications without significant application design work.

[0049] Figure 2 Shown for Figure 1 A data structure 202 for storing data nodes in the data repository 124. From a high-level perspective, the data structure 202 stores information in a hierarchical structure of nodes. In other words, the data structure 202 stores information in a tree-like file system. The bucket manager 104 can identify the data nodes in the tree by specifying a path from the root node of the data tree to the data node under consideration. The root node of the data tree identifies a specific application (here "App1"). In some environments, the data repository 124 may store multiple data structures, and the multiple data structures are associated with multiple corresponding applications (e.g., "App1", "App2", "App3", etc.) with multiple corresponding root nodes. This allows the bucket manager 104 to simultaneously manage bucket allocations for two or more applications running on the same computing device.

[0050] The first data portion 204 of the data structure 202 belongs to the "machine" data branch. The data nodes emanating from this branch identify the computing devices included in the distributed system at any given time. More specifically, each computing device creates a data node in the first data portion 204 when it joins the distributed system. The storage and retrieval component 120 also automatically removes any data node in the first data portion 204 when the computing device associated with the data node shuts down, fails, or otherwise stops serving the particular application session under consideration. In this sense, the data nodes in the first data portion 204 can be considered to be transient or non-persistent.

[0051] In addition, when creating this type of node, the storage and retrieval component 120 appends data in the first data portion 204 that identifies the temporal position of the node relative to other nodes. The bucket manager 104 can use the order information associated with the data nodes in the first data portion 204 to identify the temporal order in which computing devices joined the distributed system. The bucket manager 104 can in turn use this insight to determine whether it should assume the role of leader at any given time. (Additional information about this behavior is described below.) Figure 3 .)

[0052] The second data portion 206 of the data structure 202 is associated with a "Machine2Buckets" data path that, given a specified computing device, provides information about the buckets assigned to that particular computing device. In other words, the "Machine2Buckets" data path maps the specified machine to its associated buckets. The leader computing device creates and modifies the data nodes in the second data portion 206. The follower computing device can consult the second data portion 206 to determine which buckets have been assigned to it. That is, if the follower computing device corresponds to computing device X, the device consults the data node associated with device X in the second data portion 206 to find out which buckets are assigned to it. When the computing device is shut down, fails, or otherwise stops serving the application session under consideration, the storage and retrieval component 120 retains this information for the computing device in the second data portion 202. In this sense, the data nodes in the second data portion 206 can be considered non-transient or persistent. Unlike the first data portion 204, the data nodes of the second data portion 206 do not capture the order in which the data nodes were created.

[0053] The third data portion 208 of the data structure 202 stores data nodes under a "Bucket2Machine" data path that provides information about what computing device is currently claiming ownership of the bucket given a specified bucket. In other words, the "Bucket2Machine" data maps the specified bucket to the machine that currently owns the bucket. Each computing device that claims ownership of the bucket can create a data node in the third data portion 208 to remember the event. Each computing device that relinquishes control of the bucket can remove the corresponding data node in the third data portion 208. Before claiming ownership of the bucket, the computing device can also consult the third data portion 208 to determine whether the bucket is currently owned by another computing device. Similar to the first data portion 204, the data nodes in the third data portion 208 can be considered to be transient or non-persistent. Unlike the first data portion 204, the data nodes in the third data portion 208 do not capture the time order in which the data nodes were created.

[0054] Figure 3 An example is provided of how bucket managers associated with three computing devices (302, 304, 306) utilize data nodes in the data repository 124 to determine what bucket manager should operate as a leader. These data nodes specifically correspond to the ephemeral ordered data nodes provided by the first data portion 204.

[0055] In operation, the bucket manager of the first computing device X 302 creates a first sequential data node 308 when it is initialized. The bucket manager of the second computing device Y 304 creates a second sequential data node 310 when it is initialized. The bucket manager of the third computing device Z 306 creates a third sequential data node 312 when it is initialized. The three data nodes (308, 310, 312) include information identifying their relative order. In addition, each bucket manager registers to monitor any data node that immediately precedes its own data node. For example, when any changes occur to the data node 310 associated with the second computing device Y 304, the third computing device Z 306 registers to receive notifications. Upon discovering that data node 310 has been removed (for example, because the second computing device Y 304 has failed or is otherwise shut down), the bucket manager of the third computing device Z 306 will register to monitor the immediately preceding data node - in this case, data node 308. Finally, the bucket manager that discovers that it has no preceding data node will assume the role of leader. For example, the bucket manager of the first computing device X 302 determines that it has the oldest created data node 308, so it should assume the role of leader.

[0056] Figure 4 Shows the use of Figure 1An illustrative distributed system 402 of logical components. More specifically, the distributed system 402 includes three representative computing devices (404, 406, 408), although some environments may include tens, hundreds, thousands, etc. of computing devices. Each computing device may correspond to a server or any other type of computing device. Section C describes representative hardware that each computing device may include. Each computing device also interacts with the S&N system 108. The S&N system 108 itself may be implemented using one or more computing devices (e.g., servers). In the case where the S&N system 108 uses multiple computing devices, it may include a mechanism for replicating information in the data repository 124 across multiple computing devices. The computing devices (404, 406, 408) may interact with the S&N system 108 via a point-to-point link, a local area network, a wide area network, etc., or any combination thereof.

[0057] Each computing device includes one or more application instances and a bucket manager. For example, computing device X404 includes application instance A 410 and application instance B 412 that are communicatively coupled to a bucket manager 414. Each application instance performs application operations specific to the environment. In addition, each application instance loads a set of initialization parameters to the bucket manager 414. Similarly, computing device Y 406 includes application instance A 416, application instance B 418, and a bucket manager 420. Computing device Z408 includes application instance A 422, application instance B 424, and a bucket manager 426. Bucket managers (414, 420, 426) operate in different independent allocation spaces when interacting with different corresponding applications. However, for ease of explanation, the remaining description will assume that the bucket managers (414, 420, 426) only work in collaboration with a single application such as App A.

[0058] At any given time, only one bucket manager operates as a leader; the other bucket managers operate as followers. In addition, at any given time, each application instance can perform its tasks against a set of buckets. The set of buckets is disjoint on the computing devices (relative to the same application). This means that no two computing devices are assigned the same bucket at any given time (relative to the same application). This avoids errors that can occur when two or more computing devices perform duplicate work for the same task.

[0059] Figures 5 to 7 shows the method for routing tasks to Figure 4 Three illustrative working routing components (502, 602, 702) of buckets in a distributed system. Figure 5Initially, the hash component 504 generates a hash (e.g., but not limited to, a 128-bit hash value) of the data associated with the task. For example, consider again the case where an application performs processing on data associated with a website. The hash component 504 may generate a hash of a URL associated with the website and / or a page content associated with the website, etc. The hash component 504 may use any hash algorithm to perform this task, as long as the hash algorithm evenly distributes the hashes over the hash space. Illustrative hash algorithms that may be used include the MD5 algorithm (e.g., described in RFC1321 (1992)), the SHA family algorithm (e.g., described in RFC3174 (2001)), etc. To provide only one non-limiting example, the hash component 504 may apply various environment-specific rules to normalize the URL, append random information to it (by "salting" it), and then use the MD5 algorithm to process the resulting information to generate a hash. The hash component 504 may then extract the n most significant bits of the hash to generate a bucket ID associated with the task under consideration. The bucket ID identifies a specific segment of the hash space. The bucket ID also identifies a specific bucket.

[0060] The work management component 506 uses the bucket ID to forward the task request to the computing device to which the specific bucket is currently assigned. For example, assume that the second computing device Y 406 currently owns the specific bucket associated with the bucket ID. The work management component 506 can forward the task request to computing device Y 406. The application instance running on the second computing device Y 406 then obtains any data required to perform the task and then performs the task itself. The above behavior assumes that the work management component 506 has access to information provided by the data repository 124 that specifies the association between the computing device and the bucket.

[0061] exist Figure 6 In the example of , assume that the work management component 604 does not initially know what computing device owns the particular bucket. The work management component 604 can handle this scenario by first forwarding the task request to any randomly selected computing device, such as the second computing device Y 406, etc. The work management component 604 can then rely on the second computing device Y 406 to forward the task request to the computing device that actually owns the particular bucket (assuming that the second computing device Y 406 does not currently own the particular bucket). In this case, the second computing device Y 406 forwards the request to the third computing device Z 408. This behavior assumes that the second computing device Y 406 can use the information provided in the data repository 124 to determine the computing device that owns the particular bucket.

[0062] exist Figure 7In the example of , it is again assumed that the work management component 704 does not initially know what computing device owns the particular bucket. And it is again assumed that the work management component 704 first forwards the task request to the second computing device Y 406. In this case, the second computing device Y 406 identifies the computing device that owns the sought-after bucket and returns this information to the work management component 704. The work management component 704 uses this information to send the task request to the appropriate computing device, which here corresponds to the third computing device Z 408.

[0063] In the above example, when sending a task request to the target computing device that currently owns the bucket, there is an opportunity to reallocate the target bucket. The target computing device can resolve this scenario by automatically forwarding the task request to the correct computing device. Alternatively, the target computing device can mark the task request as failed, whereupon the work management component can repeat the above routing process.

[0064] The above three examples of work routing components are described in the spirit of illustration and not limitation. Other implementations may use other strategies to forward tasks to appropriate computing devices.

[0065] Figure 8 Shows the use of Figure 1 The logical components of the system 802 implement a distributed system of upload services. In this environment, a crawler system 804 crawls the Internet 806 to discover new websites, or newly modified websites (compared to the immediately preceding crawler operation). The crawler system 804 stores data entries about each such identified website in a data repository 808. The work routing component 810 then sends one or more task requests (812, 814, 816) to a distributed system including multiple upload servers. In one case, each such task request instructs the upload server to process the following website data, which is associated with a single URL mapped to a bucket owned by the upload server. In another case, each task request instructs the upload server to process a batch of websites, which have corresponding URLs mapped to buckets owned by the upload server. Upon receiving such a request, the upload server pulls the appropriate data from the data repository 808 and processes it. More specifically, each upload server converts the data associated with the website into an inverted index format that can be used by the distributed indexing system 818. The upload server then transmits the processed data to the distributed indexing system 818. A search engine (not shown) uses distributed indexing system 818 to map user-submitted query terms to websites containing those query terms.

[0066] The upload servers (812, 814, 816) include respective bucket managers (BMs) (820, 822, 824) that collectively assign buckets to the upload servers (812, 814, 816). To perform this task, each bucket manager interacts with the S&N system 108. As described above, at any given time, only one bucket manager operates as a leader. In addition, at any given time, no two upload servers have the same bucket.

[0067] In the upload service, bucket allocation is "expensive" in that it requires considerable time and resources to perform and therefore adversely affects system throughput. For example, once a new bucket is allocated, the upload service needs to parse various files to determine the status of the upload process for the job associated with the bucket. The upload service also needs to identify and remove temporary files left behind by the previous owner of the bucket, etc. The techniques described here help reduce these expensive bucket reallocations.

[0068] Figure 8 The examples are presented in the spirit of illustration and not limitation. Other implementations may use Figure 4 The architecture shown can be used to perform any other application tasks.

[0069] B. Exemplary Procedure

[0070] Fig. 9 18 shows in the form of a flow chart the process of explaining the operation of the distributed system of Part A. Since the principles underlying the operation of the distributed system have been described in Part A, some operations will be addressed in a general manner in this section. As noted in the preamble to the detailed description, each flow chart is represented as a series of operations performed in a particular order. However, the order of these operations is merely representative and may be changed in any manner.

[0071] In addition, it will be combined Figure 2 202 Reference to Data Structures Figure 4 Distributed Systems 402 explained Fig. 9 -18. To simplify the description, these figures refer to a particular computing device in the distributed system 402 as computing device X. Computing device X has a bucket manager X. BM And application examples X APP For example, refer to Figure 4 , computing device X may correspond to computing device X 404, bucket manager X BM Can correspond to bucket manager 414, and application instance X APP It may correspond to application instance A410.

[0072] Fig. 9 It shows that when computing device X starts, bucket manager XBM The process 902 is executed. More specifically, in block 904, computing device X starts. In block 906, bucket manager X BM Interacts with the S&N system 108 to register monitoring paths. More specifically, the bucket manager X BM Register to receive notifications of changes to allocation information related to computing device X under the "Machine2Buckets" path (in the second data portion 206). Bucket Manager X BM Information for all buckets is also received under the "Bucket2Machine" data path (in the third data portion 208). Given a specified bucket, this information reveals which computing device owns the bucket. In block 908, bucket manager X BM A transient ordered node is created under the “machine” data path (in the first data portion 204 ), registering its existence and the time it joined the distributed system 402 .

[0073] In block 910, bucket manager X BM Determines if it is the leader. BucketManagerX BM This task may be performed by determining whether its transient ordered node is the first created node in the first data portion 204. If so, the bucket manager X of computing device X BM In block 912, bucket manager X BM Then, notifications are received when any data nodes in the first data portion 204 change. For example, these nodes will change when a new computing device joins the distributed system 402, or when an existing computing device leaves the distributed system 402. Alternatively, if block 910 is answered in the negative, computing device X assumes the role of a follower. (Although not in Fig. 9 , but the leader computing device also operates as a follower computing device).

[0074] Figures 10 to 12 Bucket manager X of computing device X is shown BM Actions to perform when assuming the role of leader. Fig.10 Initially, the figure shows a process 1002 for initiating an update to a distribution of computing devices in a distributed system 402. In block 1004, bucket manager X BM Determine if there are any changes to the computing devices in the distributed system 402. Bucket Manager X BM When it receives notification from the S&N system 108 (under the “machine” path) that there is a change, one or more data nodes in the first data portion 204 are notified of the change.

[0075] Assume bucket manager X BMSpecifically determine that a new computing device has joined the distributed system. If so, then in block 1006, bucket manager X BM Registration has a time period Δ T1 This time period corresponds to the monitoring of application instance X APP In Bucket Manager X BM Parameters set during initialization. In block 1008, bucket manager X BM Determine whether the new computing device subsequently leaves the distributed system 402. If this event occurs within a time period Δ defined by the bucket manager 414 T1 If the time period expires and the new computing device is not removed, bucket manager 414 will ignore the new computing device. On the other hand, in block 1010, if the time period expires and the new computing device is not removed, bucket manager X BM The bucket allocation to the computing device will start to be updated. That is, at this time, the bucket manager X BM New computing devices are considered stable.

[0076] Alternatively, assume that bucket manager X BM In block 1004, it is determined that a computing device has left the distributed system 402. If so, then in block 1012, bucket manager X BM Registration has a time period Δ T2 In block 1014, bucket manager X BM Determine whether the computing device that has left has rejoined the distributed system 402. If this event occurs in a BM During the defined time period, bucket manager X BM The departure of the computing device will be ignored. In block 1010, if the time period expires and there is no reunion of the computing device that has left the distributed system 402, the bucket manager X BM The bucket assignments to the compute devices will begin to be updated.

[0077] Fig.11 Process 1102 is shown, where when a new computing device joins the distributed system 402, bucket manager X BM Update allocation. In block 1104, bucket manager X BM Determine if there are any unassigned buckets. If so, then in block 1106, bucket manager X BM A number of buckets may be allocated to the new computing device that corresponds to the preferred number of buckets specified in the initialization parameter set. BM Determine whether the computing device can accept more buckets, but has no more unallocated buckets to give it. If so, then in block 1110, bucket manager X BMBuckets can be removed from one or more other computing devices and reallocated to the newly added computing device. BM Buckets may be removed from at least one existing computing device that has more than its preferred number and reallocated to the newly added computing device.

[0078] Fig.12 Process 1202 is shown, in which when an existing computing device leaves the distributed system 402, bucket manager X BM Update allocation. In block 1204, bucket manager X BM One or more buckets from the departing computing device are assigned to one or more existing computing devices having the least number of buckets, or otherwise representing the greatest need for additional buckets.

[0079] Notice, Fig.11 and Fig.12 The bucket allocation strategy described in is set forth by way of illustration and not limitation. Other implementations may use other strategies to allocate buckets to machines as computing devices are introduced and removed from distributed system 402. For example, in another case, a leader computing device may allow a subset of computing devices to not receive buckets. Distributed system 402 may rely on these unloaded computing devices when a specified failure condition occurs in distributed system 402 (e.g., a service metric falls below a specified threshold).

[0080] Fig.13 and Fig.14 Bucket manager X of computing device X is shown BM Actions performed when working in a follower role. More specifically, Fig.13 Shows when Bucket Manager X BM Bucket manager X discovers that it has been allocated at least one new bucket by the leader BM The leader may correspond to a bucket manager of another computing device (eg, bucket manager Y of computing device Y). BM ) or bucket manager X of the same computing device X BM .

[0081] In block 1304, bucket manager X BM Receives a notification from the S&N system 108 informing it that a new bucket has been assigned to its computing device. It receives this notification because it had previously registered a watch on the "Machine2Buckets" data path of computing device X. At block 1306, Bucket Manager X BMDetermine if the allocated bucket is currently owned by another computing device. It performs this task by querying the "Bucket2Machine" data path to discover if it contains any data. Assume that the answer to block 1306 is yes. In block 1308, bucket manager 414 registers a watch on the "Bucket2Machine" data path. In block 1310, bucket manager X BM Receive notification that the device that previously claimed ownership of the bucket has relinquished that ownership. In block 1312, Bucket Manager X rechecks the "Machine2Buckets" data path. BM Confirm that it is still assigned the bucket in question.

[0082] At this time, the bucket manager X BM Be prepared to assume ownership of the newly allocated bucket. In block 1314, Bucket Manager X BM Use the addbucket callback to add the application instance X associated with computing device X. APP Send out the following notification: it should start processing the task of the newly allocated bucket. In block 1316, bucket manager X BM Ownership of the newly allocated bucket is formally claimed by setting the appropriate data nodes in the third data portion 208 (ie, under the "Bucket2Machine" data path).

[0083] Fig.14 The bucket manager X of the follower computing device X is shown BM Process 1402 is performed when removing a bucket assigned to it. In block 1404, bucket manager X BM Receive a notification under the "Machine2Buckets" data path indicating that a bucket previously assigned to it has been removed. In block 1406, Bucket Manager X BM Respond by notifying application instance X running on computing device X APP Remove the previously allocated bucket. In block 1408, Bucket Manager X BM Ownership of the removed bucket is formally relinquished by removing the appropriate ephemeral data node in the third data portion 208 under the "Bucket2Machine" data path.

[0084] Fig.15 and 16 Shown by application example X APP Upon receiving notification of a new bucket (relative to Fig.15 ) or removal of an existing bucket (relative to Fig.16 ). More specifically, Fig.15 The process for assigning a new bucket to an application instance X is shown. APPIn block 1504, application instance X APP From its associated bucket manager X BM Receive notification that a new bucket has been assigned to it. In block 1506, application instance X APP Load appropriate resources that enable it to perform the tasks associated with the newly allocated bucket. In block 1508, application instance X APP The tasks are executed (eg, when fed by a work management component).

[0085] Fig.16 shows an example for processing by application X APP The process 1602 of removing the existing bucket. In block 1604, apply instance X APP From its associated bucket manager X BM Receive notification that an existing bucket has been removed. In block 1606, application instance X APP Remove the resources that were previously used to process tasks associated with the now-removed bucket.

[0086] Fig.17 and Fig.18 The process 1702 is shown together, which provides the bucket manager X BM A summary of the operations performed in its leader and follower roles. Assume again that bucket manager X BM Associated with computing device X.

[0087] In block 1704, process 1702 involves performing a APP Receive initialization parameters to initialize bucket manager X BM Bucket Manager X BM Corresponding to the generic component, it can be reconfigured by different initialization parameter sets to work with different application instances. In blocks 1706 and 1708, bucket manager X BM Interact with the S&N system 108 to determine whether computing device X 404 is to be assigned the role of a leader computing device.

[0088] When operating in the leader role, in block 1710, bucket manager X BM Interact with the S&N system 108 to determine if there are any changes in the membership of a group of computing devices included in the distributed system 402. In block 1712, when a membership change occurs, the bucket manager X BM Ensure that the change is stable. In block 1714, when the change is determined to be stable, bucket manager X BMUpdate the allocation of buckets to computing devices in the distributed system 402 so that no two computing devices are assigned the same bucket, each bucket corresponding to a quota of work to be performed. In block 1716, bucket manager X BM Interacts with the S&N system 108 to record allocations of buckets.

[0089] When operating in the Follower role, Fig.18 In block 1802, bucket manager X BM Interact with the S&N system 108 to determine if there is a change in the bucket assigned to computing device X. In block 1804, when there is a change in the bucket, bucket manager X BM To application instance X APP Notify the bucket of the change. In block 1806, bucket manager X BM Interacts with the S&N system 108 to request any new BM relinquish ownership of the newly removed buckets, and / or relinquish ownership of any newly removed buckets.

[0090] C. Representative computing devices

[0091] Fig.19 A computing device 1902 is shown that can be used to implement any aspect of the mechanisms described in the above figures. For example, referring to Figure 4 , Fig.19 The type of computing device 1902 shown may be used to implement any computing device (404, 406, 408) in distributed system 402, or any computing device used to implement S&N system 108, etc. In all cases, computing device 1902 represents a physical and tangible processing mechanism.

[0092] The computing device 1902 may include one or more hardware processors 1904. The hardware processors may include, but are not limited to, one or more central processing units (CPUs), and / or one or more graphics processing units (GPUs), and / or one or more application specific integrated circuits (ASICs), etc. More generally, any hardware processor may correspond to a general-purpose processing unit or a special-purpose processor unit.

[0093] The computing device 1902 may also include computer-readable storage media 1906 corresponding to one or more computer-readable media hardware units. The computer-readable storage media 1906 stores any kind of information 1908, such as machine-readable instructions, settings, data, and the like. For example, without limitation, the computer-readable storage media 1906 may include one or more solid-state devices, one or more magnetic hard disks, one or more optical disks, magnetic tapes, and the like. Any instance of the computer-readable storage media 1906 may use any technology for storing and retrieving information. Furthermore, any instance of the computer-readable storage media 1906 may represent a fixed or removable unit of the computing device 1902. Furthermore, any instance of the computer-readable storage media 1906 may provide for volatile or non-volatile retention of information.

[0094] The computing device 1902 may utilize any instance of the computer-readable storage medium 1906 in different ways. For example, any instance of the computer-readable storage medium 1906 may represent a hardware storage unit (such as random access memory (RAM)) for storing transient information during execution of a program by the computing device 1902, and / or a hardware storage unit (such as a hard disk) for retaining / archiving information on a more permanent basis. In the latter case, the computing device 1902 also includes one or more drive mechanisms 1910 (such as a hard disk drive mechanism) that retrieve information from the instance of the computer-readable storage medium 1906.

[0095] Computing device 1902 may perform any of the functions described above when hardware processor 1904 executes computer-readable instructions stored in any instance of computer-readable storage medium 1906. For example, computing device 1902 may execute computer-readable instructions to perform each block of the process described in Section B.

[0096] Alternatively or additionally, the computing device 1902 may rely on one or more other hardware logic units 1912 to perform operations using task-specific collections of logic gates. For example, the hardware logic unit 1912 may include a fixed configuration of hardware logic gates, such as created and set at the time of manufacturing and cannot be changed thereafter. Alternatively or additionally, the other hardware logic units 1912 may include a collection of programmable hardware logic gates that can be set to perform different dedicated tasks. The latter class of devices includes, but is not limited to, programmable array logic devices (PALs), general array logic devices (GALs), complex programmable logic devices (CPLDs), field programmable gate arrays (FPGAs), etc.

[0097] Fig.19The hardware logic circuitry 1914 is generally indicated to include any combination of a hardware processor 1904, a computer-readable storage medium 1906, and / or other hardware logic units 1912. That is, the computing device 1902 may employ any combination of a hardware processor 1904 that executes machine-readable instructions provided in the computer-readable storage medium 1906 and / or one or more other hardware logic units 1912 that perform operations using fixed and / or programmable sets of hardware logic gates. More generally, the hardware logic circuitry 1914 corresponds to any type of one or more hardware logic units that perform operations based on logic stored in and / or otherwise included in the hardware logic unit.

[0098] In some cases (e.g., where computing device 1902 represents a user computing device), computing device 1902 also includes input / output interfaces 1916 for receiving various inputs (via input devices 1918) and for providing various outputs (via output devices 1920). Illustrative input devices include a keyboard device, a mouse input device, a touch screen input device, a digitizer tablet, one or more still image cameras, one or more video cameras, one or more depth camera systems, one or more microphones, a voice recognition mechanism, any movement detection mechanism (e.g., accelerometers, gyroscopes, etc.), etc. A specific output mechanism may include a display device 1922 and an associated graphical user interface presentation (GUI) 1924. Display device 1922 may correspond to a liquid crystal display device, a light emitting diode display (LED) device, a cathode ray tube device, a projection mechanism, etc. Other output devices include a printer, one or more speakers, a tactile output mechanism, an archiving mechanism (for storing output information), etc. The computing device 1902 may also include one or more network interfaces 1926 for exchanging data with other devices via one or more communication conduits 1928. One or more communication buses 1930 communicatively couple the above-mentioned units together.

[0099] The communication conduit 1928 may be implemented in any manner, such as through a local area computer network, a wide area computer network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. The communication conduit 1928 may include any combination of hardwired links, wireless links, routers, gateway functions, name servers, etc., managed by any protocol or combination of protocols.

[0100] Fig.19 The computing device 1902 is shown as being composed of a discrete collection of discrete units. In some cases, the collection of units may correspond to discrete hardware units provided in a computing device chassis having any form factor. In other cases, the computing device 1902 may include an integrated Figure 1For example, computing device 1902 may include a system on a chip (SoC or SOC), which corresponds to a hardware logic unit that combines the functions of two or more units shown. Fig.19 An integrated circuit that incorporates the functionality of two or more units as shown.

[0101] The following summary provides a non-exhaustive collection of illustrative aspects of the techniques set forth herein.

[0102] According to a first aspect, a computing device for operation within a distributed system of computing devices is described. The computing device includes an application instance for executing a specified service. It also includes a bucket manager, which is configured to: receive a set of initialization parameters from the application instance; and interact with a storage and notification (S&N) system to determine whether the computing device will be assigned the role of a leader computing device. When operating in the leader role, the bucket manager is configured to: interact with the S&N system to determine whether there are any changes in the membership of the computing devices included in the distributed system; when the membership changes, ensure that the changes are stable; when it is determined that the changes are stable, update the allocation of buckets to the computing devices in the distributed system so that no two computing devices are allocated the same bucket, each bucket corresponding to a quota of work to be performed; and interact with the S&N system to record the allocation of buckets. When operating in the follower role, the bucket manager is configured to: interact with the S&N system to determine whether there are changes in buckets allocated to the computing device; when there are changes in buckets, notify the application instance of the change in buckets; and interact with the S&N system to claim ownership of any newly allocated buckets, and / or give up ownership of any newly removed buckets. The computing device includes hardware logic circuitry including: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and / or (b) one or more other hardware logic units that perform operations using task-specific sets of logic gates.

[0103] According to a second aspect, the specified service corresponds to: processing one or more data items associated with one or more corresponding network resources to produce one or more processed data items; and uploading the processed data items to a distributed indexing system. The search engine uses the distributed indexing system to match input queries to network resources.

[0104] According to a third aspect, an S&N system provides: a data repository for storing data nodes in a hierarchical data structure; and a notification component configured to notify a computing device of changes in the data nodes.

[0105] According to a fourth aspect related to the third aspect, a data repository includes: a first set of data nodes that identify computing devices that are currently members of a distributed system; a second set of data nodes that identify buckets currently assigned by a leader bucket manager to each computing device in the distributed system; and a third set of data nodes that identify ownership of buckets as claimed by computing devices in the distributed system. The first set of data nodes is non-persistent because these data nodes will be automatically removed when a session involving an application instance is terminated. The second set of nodes is persistent because these nodes will not be automatically removed when a session involving an application instance is terminated. The third set of nodes is also non-persistent.

[0106] According to a fifth aspect related to the fourth aspect, the first set of data nodes further comprises data associated therewith, the data indicating an order of the data nodes in the first set of data nodes relative to each other.

[0107] According to a sixth aspect, the bucket manager also interacts with another application instance that performs another specified service, and the other application instance is associated with another set of initialization parameters.

[0108] According to a seventh aspect, an initialization parameter specifies the number of buckets that a computing device preferably has.

[0109] According to an eighth aspect, each bucket is associated with a bucket identifier. The assignment of tasks to buckets is determined by: creating a hash of the data associated with the task; and mapping the hash to the bucket identifier associated with the particular bucket.

[0110] According to the ninth aspect, in the leader role, the bucket manager responds to a determination that a new computing device has been added by ensuring that the change is stable by: waiting for at most a specified amount of time to determine whether the new computing device has been removed, the specified amount of time being specified by a set of initialization parameters; and if the new computing device is not removed within the specified amount of time, identifying the new computing device as stable.

[0111] According to the tenth aspect, in the leader role, the bucket manager responds to a determination that an existing computing device has been removed by ensuring that the changes are stable by: waiting for at most a specified amount of time to determine whether the existing computing device returns to the computing device group, the specified amount of time being specified by a set of initialization parameters; and if the existing computing device does not return within the specified amount of time, specifying that the existing computing device has left the group.

[0112] According to the eleventh aspect, in the follower role, the bucket manager responds to a determination that a new bucket has been assigned to it by: interacting with the S&N system to determine whether the new bucket is currently owned by another computing device; and when the new bucket is owned by another computing device, waiting until the S&N system indicates that the other computing device has relinquished ownership of the new bucket before claiming ownership of the new bucket.

[0113] According to the twelfth aspect, a method for managing the operation of a computing device within a distributed system of computing devices is described. The method includes: initializing a bucket manager based on a set of initialization parameters received from an application instance installed on the computing device, the bucket manager corresponding to a common resource, and the common resource can be reconfigured by different sets of initialization parameters to work in conjunction with different application instances; and using the bucket manager to interact with a storage and notification (S&N) system to determine whether the computing device will be assigned the role of a leader computing device. When operating in the leader role, the bucket manager operates to: interact with the S&N system to determine whether there are any changes in the membership of a group of computing devices included in the distributed system; when the membership changes, ensure that the change is stable; when it is determined that the change is stable, update the allocation of buckets to computing devices in the distributed system so that no two computing devices are allocated the same bucket, each bucket corresponds to a quota of work to be performed, and determine the allocation of tasks to buckets by creating a hash of data associated with the task and mapping the hash to a bucket identifier associated with a specific bucket; and interact with the S&N system to record the allocation of buckets. When operating in the follower role, the bucket manager operates by: interacting with the S&N system to determine whether there are changes to buckets assigned to the computing device; notifying the application instance of the bucket changes when there are changes to the buckets; and interacting with the S&N system to claim ownership of any newly assigned buckets and / or relinquish ownership of any newly removed buckets.

[0114] According to the thirteenth aspect related to the twelfth aspect, in the leader role, the bucket manager responds to the determination that a new computing device has been added by ensuring that the change is stable by: waiting for at most a specified amount of time to determine whether the new computing device is removed, the specified amount of time being specified by a set of initialization parameters; and if the new computing device is not removed within the specified amount of time, identifying the new computing device as stable.

[0115] According to a fourteenth aspect related to the twelfth aspect, in a leader role, the bucket manager responds to a determination that an existing computing device has been removed by ensuring that changes are stable by: waiting for at most a specified amount of time to determine whether the existing computing device returns to the computing device group, the specified amount of time being specified by a set of initialization parameters; and if the existing computing device does not return within the specified amount of time, specifying that the existing computing device has left the group.

[0116] According to a fifteenth aspect related to the twelfth aspect, in a follower role, the bucket manager responds to a determination that a new bucket has been assigned to it by: interacting with the S&N system to determine whether the new bucket is currently owned by another computing device; and when the new bucket is owned by another computing device, waiting until the S&N system indicates that the other computing device has relinquished ownership of the new bucket before claiming ownership of the new bucket.

[0117] According to a sixteenth aspect, a distributed system is described, comprising: a plurality of computing devices; and a storage and notification (S&N) system, the S&N system being configured to store data nodes in a hierarchical data structure. Each computing device includes a bucket manager and an application instance. The bucket manager of each computing device is configured to: receive a set of initialization parameters from the application instance; and interact with the S&N system to determine whether the computing device will be assigned the role of a leader computing device. When operating in a leader role, the bucket manager is configured to: interact with the S&N system to determine whether there are any changes in the membership of the computing devices included in the distributed system; when the membership changes, ensure that the change is stable; when it is determined that the change is stable, update the allocation of buckets to the computing devices in the distributed system so that no two computing devices are assigned the same bucket, each bucket corresponding to a quota of work to be performed; and interact with the S&N system to record the allocation of buckets. When operating in a follower role, the bucket manager is configured to: interact with the S&N system to determine if there is a change in buckets assigned to the computing device; when there is a change in buckets, notify the application instance of the change in buckets; and interact with the S&N system to claim ownership of any newly assigned buckets, and / or relinquish ownership of any newly removed buckets. Each computing device includes hardware logic circuitry, the hardware logic circuitry including: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in memory, and / or (b) one or more other hardware logic units that perform operations using a task-specific set of logic gates.

[0118] According to a seventeenth aspect related to the sixteenth aspect, in a leader role, the bucket manager responds to a determination that a new computing device has been added by ensuring that the change is stable by: waiting for at most a prescribed amount of time to determine whether the new computing device has been removed, the prescribed amount of time being specified by a set of initialization parameters; and if the new computing device has not been removed within the prescribed amount of time, identifying the new computing device as stable.

[0119] According to an eighteenth aspect related to the sixteenth aspect, in a leader role, the bucket manager responds to a determination that an existing computing device has been removed by ensuring that changes are stable by: waiting for at most a specified amount of time to determine whether the existing computing device returns to the computing device group, the specified amount of time being specified by a set of initialization parameters; and if the existing computing device does not return within the specified amount of time, specifying that the existing computing device has left the group.

[0120] According to the nineteenth aspect, with respect to the sixteenth aspect, the distributed system also includes a work routing component, which is configured to: receive a hash of data associated with a task; map the hash to a bucket identifier associated with a specific bucket; and route the task request to a computing device currently associated with the bucket.

[0121] According to the twentieth aspect related to the sixteenth aspect, the distributed system also includes a work routing component, which is configured to: send a task request to a specific computing device in the distributed system; and when the specific computing device does not own the bucket associated with the task request, rely on the specific computing device to route the task request to another computing device that currently owns the bucket.

[0122] The twenty-first aspect corresponds to any combination (eg, any logically consistent permutation or subset) of the above-mentioned first to twentieth aspects.

[0123] The twenty-second aspect corresponds to any method counterpart, device counterpart, system counterpart, apparatus plus function counterpart, computer-readable storage medium counterpart, data structure counterpart, product counterpart, graphical user interface presentation counterpart, etc. associated with the first to twenty-first aspects.

[0124] Finally, the functionality described herein may employ various mechanisms to ensure that any user data is processed in a manner consistent with applicable laws, social norms, and the expectations and preferences of individual users. For example, the functionality may allow users to explicitly opt-in (and then explicitly opt-out) of the functionality's provisions. The functionality may also provide appropriate security mechanisms to ensure the privacy of user data (e.g., data security mechanisms, encryption mechanisms, password protection mechanisms, etc.).

[0125] Furthermore, the present specification may set forth various concepts in the context of illustrative challenges or problems. Such an interpretation is not intended to imply that others understand and / or interpret the challenges or problems in the manner specified herein. Furthermore, such an interpretation is not intended to imply that the subject matter recited in the claims is limited to solving the identified challenges or problems; that is, the subject matter in the claims may be applied in the context of challenges or problems other than those described herein.

[0126] Although the subject matter has been described in language specific to structural features and / or methodological acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are disclosed as example forms of implementing the claims.

Claims

1. A specific computing device for operation within a distributed system of computing devices, the distributed system comprising a group of multiple computing devices, the specific computing device being a member of the group of computing devices, the specific computing device comprising: Hardware logic circuitry, the hardware logic circuitry comprising: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and / or (b) one or more other hardware logic units that use a collection of logic gates to perform the operations; an application instance implemented by the hardware logic circuit system, the application instance being configured to execute a specified service; A bucket manager implemented by the hardware logic circuit, the bucket manager being configured to perform a process for managing the operation of the specific computing device as a leader role and a follower role, the process comprising: receiving a set of initialization parameters from the application instance; interacting with a storage and notification S&N computing system over a communication conduit to determine whether the computing device is to be assigned a leader role, wherein the computing device group is connected to the S&N computing system via a communication link, the S&N computing system also being part of the distributed system, the S&N computing system acting as an intermediary enabling the computing devices in the group to indirectly coordinate their behaviors; When operating in the described leader role: interacting with the S&N computing system over the communication conduit to determine if there are any changes to membership of the group of computing devices included in the distributed system; When the membership changes, ensuring that the changes are stable; When the change is determined to be stable, updating the allocation of buckets to the group of computing devices in the distributed system so that no two computing devices in the group are allocated the same bucket, each of the buckets allocated to the group corresponding to a quota of work to be performed; and interacting with the S&N system over the communication conduit to record the assignment of buckets to the group of computing devices; and When operating in the follower role: interacting with the S&N computing system over the communication conduit to determine if there is a change to a bucket assigned to the particular computing device; When there is a change in the bucket assigned to the specific computing device, notifying the application instance of the change in the bucket assigned to the specific computing device; and Based on the change of the bucket, interact with the S&N computing system on the communication pipe to claim ownership of any bucket that has been assigned to the particular computing device, and / or based on the change of the bucket, relinquish ownership of any bucket that has been removed from the particular computing device.

2. The specific computing device of claim 1, wherein the specified service corresponds to: processing one or more data entries associated with one or more corresponding network resources to produce one or more processed data entries; and uploading the processed data items to a distributed indexing system, The search engine uses the distributed indexing system to match input queries to the one or more corresponding network resources.

3. The specific computing device of claim 1, wherein the S&N computing system provides: A data storage repository for storing data nodes in a hierarchical data structure; and A notification component is implemented by a hardware logic circuit system provided by the S&N computing system, and the notification component is configured to notify the computing device group of the change of the data node.

4. The particular computing device of claim 3, wherein the data repository comprises: a first set of data nodes identifying the group of computing devices that are currently members of the distributed system; a second set of data nodes identifying buckets currently assigned to each computing device in the group of computing devices in the distributed system; as well as a third set of data nodes that identify ownership of buckets as respectively claimed by the computing devices in the group of computing devices in the distributed system, The first group of data nodes is non-persistent because the data nodes in the first group of data nodes will be automatically removed when the corresponding application instance session in the computing device group is terminated. wherein the second set of nodes is persistent in that the nodes in the second set of data nodes will not be automatically removed when the corresponding application instance session is terminated, and The third group of nodes is also non-persistent.

5. The particular computing device of claim 4, wherein the first set of data nodes further comprises data associated therewith indicating an order of the data nodes in the first set of data nodes relative to each other.

6. The specific computing device according to claim 1, wherein the bucket manager also interacts with another application instance of the specific computing device that executes another specified service, and the other application instance is associated with another set of initialization parameters.

7. The specific computing device of claim 1, wherein an initialization parameter specifies the number of buckets that the specific computing device preferably has.

8. The specific computing device of claim 1 , wherein a specific bucket of the buckets assigned to the specific computing device is associated with a specific bucket identifier, and wherein the assignment of a specific task to the specific bucket is determined by: creating a hash of the data associated with the particular task; and The hash is mapped to the specific bucket identifier associated with the specific bucket.

9. The computing device of claim 1 , wherein said ensuring that said changes are stable comprises: When the bucket manager is determined to operate in the leader role, the bucket manager responds to a determination that another computing device has joined the computing device group by: waiting at most a prescribed amount of time after the other computing device joins the group of computing devices to determine whether the other computing device is removed from the group of computing devices, the prescribed amount of time being specified by the set of initialization parameters; as well as If the other computing device has not been removed from the group of computing devices within the specified amount of time, then the other computing device is identified as stable.

10. The particular computing device of claim 1, wherein said ensuring that said changes are stable comprises: When the bucket manager is determined to operate in the leader role, the bucket manager responds to a determination that another computing device has been removed from the computing device group by: waiting at most a prescribed amount of time after the other computing device is removed from the group of computing devices to determine whether the other computing device returns to the group of computing devices, the prescribed amount of time being specified by the set of initialization parameters; as well as If the existing computing device does not return to the group of computing devices within the specified amount of time, then the other computing device is indicated as having left the group of computing devices.

11. The specific computing device according to claim 1, wherein: When the bucket manager is determined to operate in the follower role, responding to a determination that a particular bucket has been assigned to the bucket manager by: interacting with the S&N computing system over the communication conduit to determine whether the particular bucket is currently owned by another computing device that is in the group of computing devices and is different from the particular computing device; as well as When a new bucket is owned by the other computing device in the group of computing devices, waiting until the S&N computing system indicates that the other computing device has relinquished ownership of the particular bucket before claiming ownership of the particular bucket.

12. A method for managing the operation of a particular computing device within a distributed system, the distributed system comprising a group of a plurality of computing devices, the particular computing device being a member of the group of computing devices, the method comprising: Initializing a bucket manager based on a set of initialization parameters received from an application instance, the bucket manager being implemented by a hardware logic circuit system provided by the specific computing device, the application instance being installed on the specific computing device and being implemented by the hardware logic circuit system provided by the specific computing device, the bucket manager corresponding to a general resource, the general resource being reconfigurable by different sets of initialization parameters to work in conjunction with different application instances; Using the bucket manager to perform a process for managing the operation of the specific computing device as a leader role and a follower role, the process comprising: interacting with a storage and notification S&N computing system over a communication conduit to determine whether the computing device is to be assigned the leader role, wherein the computing device group is connected to the S&N computing system via a communication link, the S&N computing system also being part of the distributed system, the S&N computing system acting as an intermediary enabling the computing devices in the group to indirectly coordinate their behaviors; When operating in the described leader role: interacting with the S&N computing system over the communication conduit to determine if there are any changes to membership of the group of computing devices included in the distributed system; When the membership changes, ensuring that the changes are stable; When the change is determined to be stable, updating the allocation of buckets to the group of computing devices in the distributed system so that no two computing devices in the group are allocated the same bucket, each of the buckets allocated to the group corresponding to a quota of work to be performed; determining an assignment of the particular task to the particular bucket of the buckets assigned to the group by creating a hash of data associated with the particular task and mapping the hash to a particular bucket identifier associated with the particular bucket; and interacting with the S&N system over the communication conduit to record the assignment of buckets to the group of computing devices; and When operating in the follower role: interacting with the S&N computing system over the communication conduit to determine if there is a change to a bucket assigned to the particular computing device; When there is a change in the bucket assigned to the specific computing device, notifying the application instance of the change in the bucket assigned to the specific computing device; and interacting with the S&N computing system over the communication conduit to claim ownership of any bucket that has been assigned to the particular computing device in accordance with the change in the bucket, and / or relinquishing ownership of any bucket that has been removed from the particular computing device in accordance with the change in the bucket; The hardware logic circuit system includes: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and / or (b) one or more other hardware logic units that use a set of logic gates to perform the operations.

13. The method of claim 12, wherein said ensuring that said changes are stable comprises: When the bucket manager is determined to operate in the leader role, the bucket manager responds to a determination that another computing device has joined the computing device group by: waiting at most a prescribed amount of time to determine whether the other computing device is removed from the group, the prescribed amount of time being specified by the set of initialization parameters; as well as If the other computing device is not removed from the group within the specified amount of time, then the other computing device is identified as stable.

14. The method of claim 12, wherein said ensuring that said changes are stable comprises: When the bucket manager is determined to operate in the leader role, the bucket manager responds to a determination that another computing device has been removed from the computing device group by: waiting at most a prescribed amount of time after the other computing device is removed from the group of computing devices to determine whether the other computing device returns to the group of computing devices, the prescribed amount of time being specified by the set of initialization parameters; as well as If the other computing device does not return to the group of computing devices within the specified amount of time, then the other computing device is indicated as having left the group of computing devices.

15. The method according to claim 12, wherein: When the bucket manager is determined to operate in the follower role, responding to a determination that another bucket has been assigned to the bucket manager by: interacting with the S&N computing system over the communication conduit to determine whether the another bucket is currently owned by another computing device that is in the group of computing devices and is different from the particular computing device; as well as When the new bucket is owned by another computing device, wait until the S&N computing system indicates that the other computing device has relinquished ownership of the other bucket before claiming ownership of the other bucket.

16. A distributed system comprising: a group of a plurality of computing devices, including a particular computing device; storing and notifying an S&N computing system configured to store data nodes in a hierarchical data structure, the S&N computing system acting as an intermediary that enables the computing devices in the group to indirectly coordinate their behavior; a communication conduit having a communication link connecting the group of computing devices to the S&N computing system; Each computing device in the computing device group includes an instance of hardware logic circuitry, the instance of the hardware logic circuitry implementing a corresponding bucket manager and a corresponding application instance; The specific bucket manager of the specific computing device is configured to perform a process for managing the operation of the specific computing device as a leader role and a follower role, the process comprising: receiving a set of initialization parameters from a particular application instance implemented by the particular computing device; interacting with the S&N computing system over a communication conduit to determine whether the computing device is to be assigned a leader role; When operating in the described leader role: interacting with the S&N computing system over the communication conduit to determine if there are any changes to membership of the group of computing devices included in the distributed system; When the membership changes, ensuring that the changes are stable; When the change is determined to be stable, updating the allocation of buckets to the group of computing devices in the distributed system so that no two computing devices in the group are allocated the same bucket, each of the buckets allocated to the group corresponding to a quota of work to be performed; and interacting with the S&N system over the communication conduit to record the assignment of buckets to the group of computing devices; and When operating in the follower role: interacting with the S&N computing system over the communication conduit to determine if there is a change to a bucket assigned to the particular computing device; When there is a change in the bucket assigned to the specific computing device, notifying the application instance of the change in the bucket assigned to the specific computing device; and interacting with the S&N computing system over the communication conduit to claim ownership of any bucket that has been assigned to the particular computing device in accordance with the change in the bucket, and / or relinquishing ownership of any bucket that has been removed from the particular computing device in accordance with the change in the bucket, Each instance of the hardware logic circuit system includes: (a) one or more hardware processors that perform operations by executing machine-readable instructions stored in a memory, and / or (b) one or more other hardware logic units that use a set of logic gates to perform the operations.

17. The distributed system of claim 16, wherein said ensuring that said changes are stable comprises: When the particular bucket manager is determined to operate in the leader role, the particular bucket manager responds to a determination that another computing device has joined the computing device group by: waiting at most a prescribed amount of time after the other computing device joins the group of computing devices to determine whether the other computing device is removed from the group, the prescribed amount of time being specified by the set of initialization parameters; as well as If the other computing device is not removed from the group within the specified amount of time, then the other computing device is identified as stable.

18. The distributed system of claim 16, wherein said ensuring that said changes are stable comprises: When the particular bucket manager is determined to operate in the leader role, the particular bucket manager responds to a determination that another computing device has been removed from the computing device group by: waiting at most a prescribed amount of time after the other computing device is removed from the group of computing devices to determine whether the other computing device returns to the group of computing devices, the prescribed amount of time being specified by the set of initialization parameters; as well as If the other computing device does not return to the group of computing devices within the specified amount of time, then the other computing device is indicated as having left the group of computing devices.

19. The distributed system of claim 16, further comprising a work routing component implemented by another instance of the hardware logic circuit system and configured to: Receive a specific hash of data associated with a specific task; mapping the particular hash to a particular bucket identifier associated with a particular bucket assigned to the particular computing device; and The task request is routed to the specific computing device.

20. The distributed system of claim 16, further comprising a work routing component implemented by another instance of the hardware logic circuitry and configured to: sending a task request to the specific computing device in the distributed system; and When the specific computing device does not own the specific bucket associated with the task request, the specific computing device is relied upon to route the task request to another computing device in the computing device group that currently owns the specific bucket.

Images