Dynamic worker reconfiguration across the entire work queue

Abstract

In some examples, a computing device may create multiple queues configured to receive data. The computing device may also create multiple worker instances, each associated with a respective queue to consume data from the respective queue. The individual worker instances receive system configuration information indicating the state of the respective queue and the state of the respective worker instance. Each of the individual worker instances is further configured to determine, based at least on the received system configuration information, whether to change the individual worker instance's association from the queue with which the individual worker instance is currently associated to a different queue. Based at least on the received system configuration information, at least one of the individual worker instances determines to change its association from a first queue to a second queue.

Description

Technical Field

[0001] This disclosure relates to the technical field of data processing.

Background Art

[0002] In a distributed work queue system, worker nodes may be scaled (e.g., the number or processing capacity is increased) to increase the processing bandwidth for the system. However, if the number of worker nodes is scaled beyond the range that the queuing system can support, the processing bandwidth may decrease. A conventional solution to this problem is thought to be to also increase the scale of the number of work queues in order to further partition the queuing system. Thus, the number of work queues and the number of worker nodes in the system can be scaled independently of each other. However, such a situation can lead to a performance degradation if the work queues themselves are not balanced (e.g., with respect to an even distribution of events), or if the worker nodes are not evenly distributed across the set of queues. Furthermore, errors in rebalancing can lead to a situation where there are isolated queues (e.g., queues to which no worker is assigned), leading to gaps in event processing.

Summary of the Invention

Means for Solving the Problems

[0003] Some implementations include a computing unit that generates a number of queues configured to receive data. The computing unit may also generate a number of worker instances, each worker instance may be associated with a queue to consume data from that queue. Each worker instance receives system configuration information indicating the state of each queue and the state of each worker instance. Furthermore, each individual worker instance is configured to determine, at least based on the received system configuration information, whether to change its association from the queue to which it is currently associated to a different queue. At least one of the individual worker instances decides to change its association from a first queue to a second queue, at least based on the received system configuration information. [Brief explanation of the drawing]

[0004] A detailed explanation is provided with reference to the attached diagram. In the diagram, the leftmost digit of the reference number indicates the diagram in which that reference number first appears. The use of the same reference number in different diagrams indicates similar or identical items or features.

[0005] [Figure 1] Figure 1 shows an exemplary architecture of a system that can scale queuing and worker services in several implementation forms.

[0006] [Figure 2] Figure 2 is a block diagram illustrating the logical configurations of queuing and consumption operations for several implementation forms.

[0007] [Figure 3] Figure 3 is a sequence diagram illustrating the exemplary process in which a PE worker instance maps itself to a queue or removes itself from a queue during startup, depending on the implementation configuration.

[0008] [Figure 4] Figure 4 is a sequence diagram illustrating an exemplary process in which PE worker instances, across several implementation forms, map to different queues and / or remove themselves from the queues during runtime.

[0009] [Figure 5] Figure 5 shows an example data structure used by self-managing algorithms in several implementation forms.

[0010] [Figure 6] Figure 6 shows an example of a data structure for state tracking in several implementation configurations.

[0011] [Figure 7] Figure 7 shows example pseudocode illustrating the current system state for several implementation configurations.

[0012] [Figure 8] Figure 8 shows an exemplary data structure that can be used by pub / sub programs in several implementation forms to track PE worker instances.

[0013] [Figure 9] Figure 9 shows an exemplary data structure illustrating the current queue type count for several implementation configurations.

[0014] [Figure 10] Figure 10 shows exemplary components of a service computing unit that may be used to implement at least some of the functions of the system described herein. [Modes for carrying out the invention]

[0015] Several implementations of this specification concern techniques and arrangements for creating a balance between tasks and workers, thereby resolving scalability issues between components within a distributed work queue system. For example, tasks (e.g., “events”) may be evenly distributed among multiple queues. Examples of events herein may include actions that affect stored data, such as write requests, put requests, delete requests, and move requests. In addition, workers that process tasks from queues may manage themselves based on a distributed algorithm for assigning themselves (along with tasks / events) to their respective queues in a manner that does not require centralized control for assigning workers to queues.

[0016] Some examples herein may employ a consistent hash algorithm to distribute incoming events evenly across the entire queue set. Furthermore, the distributed algorithm may be used to assign workers to specific partitions of work (e.g., queues). Based on the distributed algorithm, each worker can determine its own queue target without instructions from a central manager and without consultation with other workers. In addition, the examples herein may use an event mechanism to detect topological changes that result in queue reassignment. Furthermore, configuration constraints may be used to prevent invalid configurations (e.g., orphaned queues). Thus, the examples herein solve the scalability problem between components of a distributed work queue system. For example, some implementations may include a distributed scalable system in which various components can be scaled independently, such as based on the specific customer situation and / or use case.

[0017] In the implementations described herein, different types of queues and different types of workers may exist within the system. For example, worker types may be associated with queue types such that a particular worker consumes only from a particular type of queue to which that particular worker is associated. In addition, each queue described herein may have one or more workers corresponding to workers as consumers, so that queues are not isolated. Furthermore, workers may be scaled up to handle more queue events, and / or queues may be scaled up to support more workers. Otherwise, a particular queue with too many connected workers may become overburdened. Scaling up queues helps to improve the load balancing of queue workloads across message queuing units.

[0018] The computing systems described herein may employ message queuing programs, such as plug-ins or applications, for managing multiple work queues. Examples of message queuing programs used in some of the examples herein may include open-source multiprotocol message brokers such as RABBITMQ® and ACTIVEMQ®. The implementations described herein may take into account the scaling of the message queuing program itself (e.g., nodes in a queuing cluster), the queues (e.g., how many queues exist and where they reside), and the consumers (e.g., services read from the message queuing program encompassing the "application," such as a policy engine service (also referred to herein as a "PE").

[0019] In some examples, a set of queuing nodes can scale independently of each other. To simplify their design and deployment, some examples may include enforcement of certain restrictions, such as requiring the consumer count to be greater than or equal to the queue count.

[0020] An increase in the number of message queue consumers within a computing system may lead to an increase in increased event processing (e.g., the queue of a message queuing program may be drained more quickly with little backlog). Additionally, an increase in the number of queuing nodes within a computing system can lead to busy consumers (e.g., the consumer does not become idle waiting for work).

[0021] In some implementations, scaling of the system may include scaling of the message queuing program itself (e.g., the queuing nodes within a message queuing cluster), the queues (e.g., how many queues exist and where they are located), and the consumers (e.g., services that read from a message queuing program that includes an "application" such as a policy engine service PE). For example, a set of these nodes (queuing nodes and worker nodes) may scale independently of each other, but there may be enforcement of some restrictions, for example, the consumer (worker) count must be no more than the queue count, to simplify their design and deployment.

[0022] Increasing the number of message queuing consumers in the system can result in increased event processing (for example, the queues of the message queuing program may be filled more quickly when there is little backlog). Increasing the number of message queuing program nodes in the system can result in busy consumers (for example, consumers do not become idle waiting for work, and the message queuing program dispatches events at the same rate that consumers can process events). Therefore, the implementations herein may determine the optimal operating conditions of the system such that, for example, at the intersection of the message queuing program node count and the consumer node count, workflow latency is within an acceptable range and consumer processing is close to 100% utilization.

[0023] The examples herein may include queue and message persistence (e.g., via replication strategies that allow the system to continue operating without interruption of service or loss of events, even if a queuing node is lost), and consumption persistence (e.g., events dispatched to a queue are not orphaned by the loss of a consuming instance). Thus, the implementations herein can provide exchange / queue topologies with minimal interruption to event processing and without loss of events. Furthermore, the systems herein can manage notifications, event synchronization, replication, and similar functions.

[0024] In addition, through distributed coordination techniques, the consumers (policy engine worker instances) of this specification can determine for themselves which queue to consume from at any given time. In some examples, the administrator may initially set the number of queues of each type for the system, and this number of queues may then be scaled up or down. The system of this specification may further include mechanisms for the administrator to adjust any system configuration, and mechanisms for detecting and responding to changes.

[0025] In addition, the ability of administrators to declare the number of queues per type allows them to adjust resources to suit workflow and system-specific configurations / requirements. For example, the system may initially be deployed with a default number of queues per type, and after deployment and / or as needed to accommodate changes in customer workflow or load patterns, that number may be adjusted by the administrator.

[0026] In addition, in some examples, queue and worker counts within the system may be defined by an automated process. For example, the process may collect metrics regarding queue and worker performance. Based on an analysis of the collected metrics, the process may determine whether workers are overutilized or underutilized, and further, whether queues are under-subscription or over-subscription. Based on the analysis, the process may adjust the system to scale up or down the queue count and / or the worker count. As an example, the process may perform scaling using one or more API calls via an application programming interface (API) adopted by the system.

[0027] For the purposes of discussion, some examples herein are described in the context of a distributed computing system, including a hybrid cloud infrastructure having queue scalability and worker scalability. For example, the computing units of an object storage system may receive objects for storage and may communicate with one or more network storage systems on the network for storing objects and providing storage services to users. Furthermore, some examples herein may include a hybrid architecture that includes a local storage system communicating with one or more remote public cloud and / or private cloud storage systems.

[0028] In addition, while some examples herein describe environments in which one or more service computing units communicate with one or more network storage systems to manage data, metadata, event notifications, etc., the implementations herein are not limited to the specific examples provided and may be extended to any of the following, as will be apparent to those skilled in the art based on the disclosure herein: other types of computing system architectures and configurations, other types of storage environments, other storage providers, other types of client configurations, other types of data and tasks, etc.

[0029] Figure 1 shows an exemplary architecture of a system 100 that can scale queuing and worker services in several implementation forms. System 100 includes multiple service computing units 102 that can communicate with or connect to one or more network storage systems 104 provided by service providers 105 through one or more networks 106. In some examples, there may be multiple different service providers 105, and each service provider 105 of a different network storage system 104 may be a separate entity unrelated to the other service providers 105. Some examples of commercial network storage service providers 105 may include Amazon Simple Storage Service (S3), Microsoft Azure, Google Cloud, IBM Cloud, and Oracle Cloud. In some examples, the network storage system 104 may be called "cloud storage" or "cloud-based storage," and in some cases may provide a storage solution that has a lower cost per gigabyte than local storage, which may also be available to the service computing units 102. In addition, in some examples, the storage provider may selectively be a private storage service provider 105, or a proprietary storage service provider 105, for example, to provide access only to specific users, entities, etc., associated with the service computing unit 102. An example of a proprietary system service provider 105 may include the configuration of the HITACHI CONTENT PLATFORM.

[0030] Furthermore, the service computing unit 102 can communicate with multiple user computing units 108 and one or more administrator computing units 110 over the network 106. The user computing units 108 and administrator computing units 110 may be any of the various types of computing units discussed further below.

[0031] In some examples, the service computing unit 102 may include one or more servers that can be embodied in various ways. For example, at least part of the program, other functional components, and data storage of the service computing unit 102 may be implemented on at least one server in one or more standalone servers, a cluster of servers, a server farm, a data center, a cloud-hosted computing service, etc., but other computer architectures may be used in addition to or instead of thereto. Further details of the service computing unit 102 are discussed below with respect to Figure 10.

[0032] In some cases, multiple service computing units 102 may be grouped into one or more groups, clusters, systems, etc., at site 111, for example. In addition, multiple sites 111 may be geographically dispersed from one another to provide data replication, disaster recovery protection, etc. Furthermore, in some cases, service computing units 102 at multiple different sites 111 may be configured to communicate securely with each other to provide coordination between multiple sites 111, for example. In other examples, one or more service computing units 102 may be located separately from the others.

[0033] One or more networks 106 may include any suitable network, including wide area networks such as the Internet, local area networks (LANs) such as intranets, cellular networks, local wireless networks such as Wi-Fi and / or short-range wireless communication such as Bluetooth®, fiber channel, optical fiber, Ethernet or any other such network, wired networks including direct wired connections, or any combination thereof. Accordingly, one or more networks 106 may include wired and / or wireless communication technologies. The components used for such communication may depend at least in part on the type of network, the environment selected, or both. Protocols for communication over such networks are well known and are not described in detail herein. Accordingly, the service computing unit 102, the network storage system 104, the user device 108, and the administrator device 110 can communicate over one or more networks 106 using wired or wireless connections, or a combination thereof.

[0034] In addition, the service computing units 102 may communicate with each other over one or more networks 107. In some cases, the one or more networks 107 may be LANs, private networks, etc., and in other cases, the one or more networks 107 may include any of the networks 106 discussed above.

[0035] The service computing unit 102 may be configured to provide storage and data management services to users 112 via user devices 108, respectively. In some non-limiting examples, users 112 may include users performing functions for businesses, corporations, organizations, government agencies, academic institutions, etc., and such functions may, in some examples, include storing very large amounts of data. Nevertheless, the implementations of this specification are not limited to specific uses or applications of the system 100 and other systems and arrangements described herein.

[0036] Each user device 108 may be any suitable type of computing device, such as a desktop, laptop, tablet computing device, mobile device, smartphone, wearable device, terminal, and / or any other type of computing device capable of transmitting data over a network. Users 112 may be associated with the user device 108, for example, through their respective user accounts, user login credentials, etc. Furthermore, the user device 108 may be able to communicate with the service computing unit 102 through one or more networks 106, through separate networks, or through any other suitable type of communication connection. Many other variations will be apparent to those skilled in the art who have an interest in the disclosures herein.

[0037] Furthermore, each user device 108 may include an instance of a user application 114 that may run on the user device 108, such as for communicating with a user web application 116 that can be run on the service computing unit 102. For example, the user application 114 may be configured to transmit user data (e.g., data objects) via data instructions 118, etc., for storage in one or more network storage systems 104 and / or for receiving stored data from the network storage systems 104. In some cases, the application 114 may include a browser and may operate through a browser, and in other cases, the application 114 may include any other type of application having communication capabilities that enable communication with the user web application 116 over one or more networks 106.

[0038] In some examples, user 112 may store data in and receive data from the service computing unit 102 with which each of its user devices 108 communicates. Thus, the service computing unit 102 may provide storage for user 112 and each of its user devices 108. During steady-state operation, there may be users 108 that communicate periodically with the service computing unit 102.

[0039] In some examples, the data instructions 118 herein may be received by the service computing unit 102 from the user device 108. For example, each data instruction 118 may correspond to one or more “events”. In addition, event notifications regarding events may be generated by the service computing unit 102 for at least some of the events consumed from the queue and forwarded to one or more application computing units (not shown in Figure 1), which receive event notifications such as logging, tracking, and / or managing specific activities performed with respect to and / or data stored by the service computing unit 102.

[0040] In addition, the administrator device 110 may be any suitable type of computing device, such as a desktop, laptop, tablet computing device, mobile device, smartphone, wearable device, terminal, server, and / or any other type of computing device capable of transmitting data over a network. For example, administrators 121 may be associated with each administrator device 110 through their respective administrator accounts, administrator login credentials, etc. Furthermore, the administrator device 110 may be able to communicate with the service computing device 102 through one or more networks 106, 107, through separate networks, or through any other suitable type of communication connection.

[0041] Furthermore, each administrator device 110 may include each instance of an administrator application 122 that can be run on the administrator device 110, such as communicating with an administrator web application 124 that can be run on the service computing device 102, for example, to send management commands for managing the system 100, and / or to send management data to be stored in the network storage system 104 via management commands 126 or similar, and / or to receive management data stored from the network storage system 104. In some cases, the administrator application 122 may include a browser and operate through a browser, and in other cases, the administrator application 122 may include any other type of application having communication capabilities that enable communication with the administrator web application 124 over one or more networks 106.

[0042] In addition, the service computing unit 102 may run a management program 127 that can perform specific management tasks. For example, the management program 127 may perform operations including receiving and verifying application programming interface (API) requests received via one or more APIs 128, such as queue counts (for example, from the management device 110 if a management user determines the queue count). Thus, in some examples, the management program 127 may process management API requests received via APIs 128, such as from the management device 110. In addition, in some examples, if the management program 127 automatically performs load balancing decisions for the system 100, the management program 127 may perform other management tasks, such as creating, deleting, or otherwise managing queues and / or policy engine worker instances.

[0043] The management program 127 may also interact with a publication / subscription (pub / sub) program 129 that can notify all participants of changes to the queue count and / or worker count, so that each policy engine worker instance can independently decide whether or not to reconfigure its respective queue mapping. An example of a suitable pub / sub program is APACHE ZOOKEEPER™, but any persistent distributed pub / sub program may be used in the examples herein. For example, the pub / sub program 129 may provide read / write / update using listening semantics. In some cases, the management program 127 may update the configuration information of the pub / sub program 129.

[0044] The service computing unit 102 may run a storage program 130 that can provide a gateway to the network storage system 104, for example, to transmit data for storage in the network storage system 104 and to retrieve data requested from the network storage system 104 or from local storage 132. In addition, the storage program 130 may manage the data stored by the system 100, for example, to manage data retention periods, data protection levels, data replication, etc.

[0045] The service computing unit 102 may further include a metadata data structure such as a metadata database (DB) 134, and the metadata database (DB) 134 may be divided into multiple metadata DB parts, such as partitions, and distributed across multiple service computing units 102. For example, the metadata DB 134 may be used to manage object data 136 stored in the network storage system 104 and local object data 137 stored in the local storage 132. The metadata DB 134 may contain a lot of metadata about the object data 136, such as information about individual objects, how to access individual objects, the storage protection level of the object, the storage retention period, object owner information, object size, and object type. Furthermore, the metadata gateway program 138 may manage and maintain the metadata DB 134 to update the metadata DB 134 when new objects are stored, when old objects are deleted, when objects are migrated, etc., and to provide access to the metadata within the metadata DB 134, such as to respond to requests for data access.

[0046] In addition, the service computing unit 102 may include a policy engine program 140 which may be executed to perform data synchronization with the network storage system 104. In some cases, the policy engine program 140 may be a module of the storage program 130 or the management program 127, and in other cases, the policy engine program 140 may be a separate program. The policy engine program 140 may include a self-management algorithm 141 which may be executed to self-determine whether worker instances of the policy engine program consume from queues and / or other functions, and may access the self-management algorithm 141.

[0047] In some cases, the policy engine program 140 may work in cooperation with the message queuing program 142 to synchronize object data 136 with the network storage system 104. As mentioned above, examples of the message queuing program 142 that may be used in some examples of the system 100 herein may include open-source multiprotocol message brokers such as RABBITMQ® or ACTIVEMQ®. However, the implementation is not limited to a specific message queuing program. The message queuing program 142 may support a queuing framework 144 for providing multiple queues 145 for sending data (events) to the network storage system 104.

[0048] A worker instance of the policy engine program 140 (not shown in Figure 1) may synchronize object data 136 asynchronously with the network storage system 104. For example, rather than having to perform a specific storage operation on the network storage system 104 and complete the operation before returning a report to the user, the storage program 130 may store the object data as local object data 137 in local storage 132, add metadata for the object to the local metadata database 134 via the metadata gateway program 138, and send a reply to the requesting device with a completion message, etc. Subsequently, the storage program 130 may send the data to the queue 145 and synchronize the data asynchronously with one or more network storage systems 104 via one or more data synchronization operations 146.

[0049] In some cases, the management program 127 may employ API 128 to automatically adjust the queue count within the system 100. For example, the management program 127 may collect metrics regarding queue and worker performance. Based on an analysis of the collected metrics, the management program 127 may determine whether worker instances are overutilized or underutilized, and further, whether queues are experiencing undersubscription or oversubscription. Based on the analysis, the management program 127 may scale up or down the queue count using API calls of API 128 to the message queuing program 142, and / or scale up or down the worker count using API calls of API 128 to the policy engine program 140. Alternatively, in another example, the management user 121 may manually change the queue count or worker count by employing API calls of API 128 to send API calls to the message queuing program 142 or the policy engine program 140, respectively.

[0050] The mechanism by which API 128 updates the queue count may be essentially the same whether initiated by the management program 127 or by the management user 121. If the queue count is increased, API 128 (or management program 127) may verify that the queue count of that type does not exceed the worker count of that type. If this condition is not true, API 128 may return an error to the caller and reject the request. In addition, API 128 (or management program 127) may determine the next sequence of queues to add for types that have been determined to need scaling up and may create queues for types that are being scaled up. API 128 may cause the message queuing program 142 to start the newly created queues so that the new queues can begin accepting events.

[0051] On the other hand, when reducing the queue count for a certain type of queue, API 128 (or management program 127) may determine which queue to delete (for example, one of the queues with the highest index value in order to maintain naming expectations). Furthermore, API 128 may instruct the message queuing program 142 to prevent the selected queue from accepting additional event entries and to redistribute existing queue events to one or more remaining queues of the same type. Finally, API 128 may delete the selected queue.

[0052] In addition, when the management program 127 decides to decrease the worker count, it may, based on its analysis, select worker instances that it has determined to be underutilized and terminate the selected worker instances by sending an API call to the policy engine program 140. Similarly, when increasing the worker count, the management program 127 may, by sending an API call to the policy engine program 140, start new worker instances of a selected type determined based on its analysis as discussed above. Similarly, the management user 121 may manually increase or decrease the worker count by sending instructions to the policy engine program 140, such as via one or more API calls to API 128. In some cases, multiple worker instances may run on the same physical device, such as by running different processing threads for each different worker instance on the service computing unit 102. In other cases, different worker instances may run on different virtual machines, different physical machines, or in many other arbitrary variations, as will be apparent to those skilled in the art who have an interest in the disclosure herein. Furthermore, in some examples, rather than sending API calls to the policy engine program 140 to increase or decrease the worker count, API calls may be sent to an orchestration framework for performing these operations. Examples of suitable orchestration frameworks relating to some implementations of this specification may include, but are not limited to, KUBERNETES®.

[0053] In some examples, worker instances of the policy engine program 140 may receive events asynchronously by consuming events from one of the queues 145. Each event received by or generated within the system 100 may be queued in the message queuing framework 144 for further processing. In addition, the policy engine program 140 may provide scalable services for handling asynchronous and scheduled events. For example, events polled from the messaging queue 145 may be sent to the respective queues 156 of one or more network storage systems 104. In some examples, each network storage system 104 may provide its own queuing service 160 to provide queues 156 for receiving and distributing events, event notifications, etc.

[0054] Figure 2 is a block diagram showing exemplary logical configurations 200 for queuing and consumption operations in several implementation forms. In some examples, the logical configuration 200 may correspond to the system 100 discussed above, or to any other various possible computing system architectures that would be apparent to those skilled in the art who have an interest in the disclosure herein. The logical configuration 200 may enable complex management and storage of data using resources distributed across local and cloud systems. In this example, multiple metadata gateway instances 202 may provide events 204 to a message queuing program 142 for queuing to a message queuing framework 144. As mentioned above, examples of events 204 may include actions by a user or system 100 that affect stored data (e.g., object data 136 in the exemplary system 100 in Figure 1), such as write requests, put requests, delete requests, and move requests. More generally, in some examples, queue 145 may include three or more types, including (1) data exporters that receive data from within the system and export that data to some external target, (2) data importers that receive data from outside the system and import that data into the system, and (3) notifiers that receive internal events and map them to some external event sink.

[0055] In the examples shown, the message queuing framework 144 may receive an event 204 for queuing, as shown in 206, and distribute the event 204 to a plurality of message queues 145(1), 145(2), 145(3), 145(4), ... In some examples, at least some of each message queue 145 may be of a different type from others of each message queue 145, such as by being designated to send events to a different destination 208, or by various other types of designations. In some examples, the destination 208 may be a different storage system 104, a different queue on the same storage system 104, a different computing unit, a different application, or many other arbitrary variations of a destination as would be apparent to a person skilled in the art who is interested in the disclosure herein.

[0056] The policy engine program 140 may run as multiple separate policy engine (PE) worker instances provided by the policy engine program 140, which operate to poll and consume events from queue 145, such as for each of the destinations 208, as shown in 210, in order to provide a set of containerized services. For example, a PE worker instance 212 may asynchronously pull synchronous events from queue 145 and synchronize the events in queue 145 with those of each of the destinations 208. In some implementations, the queuing framework 216 may include separate queues 145 for each of the separate destinations, such as remote target domains. As a non-limiting example, the queuing framework 144 may include a first queue 145(1) for a first network storage system 104 (not shown in Figure 2) managed by a first storage provider, a second queue 145(2) for a second network storage system 104 (not shown in Figure 2) managed by a second storage provider, a third queue 145(3) and a fourth queue 145(4) for a third network storage system 104 (not shown in Figure 2) managed by a third storage provider, and so on.

[0057] As mentioned above, each instance of the policy engine program 140 may pull data or other events 204 for synchronization from one of the queues 145 and send the data or other events to a designated destination 208 (for example, one of the network storage systems 104 discussed above, each queue on each network storage system 104, or similar). In the examples herein, a PE worker instance 212 decides which queue to bind to. Not many jobs are placed on the PE worker instances 212, as jobs are provided to the set of PE worker instances 212 via the queuing mechanism. A single queue can maintain performance across the set of PE worker instances 212. If there are too few PE worker instances 212, the queue will not be adequately serviced, and jobs will get stuck waiting to be processed. If there are too many PE worker instances 212, the queue will not be able to provide enough work to keep up with, and the PE worker instances 212 will become underutilized.

[0058] In addition, each PE worker instance 212 may include or access a self-management algorithm 141 for self-determining the functions of its PE worker instance 212. For example, the management program 127 may collect or receive system configuration information indicating the current state of the queues 145. For example, as shown in 216, the management program 127 may communicate with the message queuing program 142 to receive the current status of the message queues 145, such as how full each queue 145 is, and how many of the queues 145 are allocated to specific types of designations. Furthermore, as shown in 217, the management program 127 may communicate with the policy engine program 140 to determine the state of the PE worker instance 212.

[0059] The management program 127 may continuously provide this information to the pub / sub program 129 as policy engine configuration information 218, which the pub / sub program 129 makes available as a PE configuration stream 220. As shown in 222, each PE worker instance 212 may also be PE configuration information 218 through the policy engine configuration stream 220, and may execute a self-management algorithm 141 to determine from which queue 145 to consume events based on the received PE configuration information 218.

[0060] In some examples, the pub / sub program maintains a global state of at least two pieces of information: (1) the number of PE worker instances 212 in the system and their identities, and (2) the desired queue count (by type). The count of PE worker instances 212 may change based on scaling events (which may be directed by the management user or by the management program 127). Similarly, changes in the queue count may also be directed by the management user or the management program 127. The pub / sub program 129 may maintain this configuration information, and the PE worker instances 212 receive and use this configuration information to determine which queue to assign themselves to. When a PE worker instance 212 decides to bind itself to a selected queue, the PE worker instance may send a bind request to the message queuing program 142, for example, via an API call to API 128.

[0061] Accordingly, each PE worker instance 212 may execute its self-management algorithm 141 based on the configuration information it receives, and based on the output of the algorithm 141, it determines the queue 145 to serve. The PE worker instance 212 then binds itself to the selected queue 145, for example, by an API call to the message queuing program, and announces that it is a consumer available from the selected queue. Based on the fact that the PE worker instance 212 binds itself to the selected queue via an API call, the message queuing program 142 may begin supplying events from the selected queue 145 to the PE worker instance 212 for processing.

[0062] In the examples herein, the message queuing program 142 does not need to communicate with the pub / sub program 129, nor does it have direct insight into the configuration information maintained by the pub / sub program 129. The message queuing program 142 recognizes consumers (PE worker instances 212) through announcements (e.g., bind or unbind requests) using the API 128. Furthermore, queues 145 may be created or deleted based on instructions from the administrative user, as discussed above, or based on instructions from the administrative program 127, which monitors the state of the configuration information 218 maintained by the pub / sub program 129 and works to bring the actual state of the system closer to the desired state represented in the pub / sub program 129. As mentioned above, the creation or deletion of queues may be performed by the administrative user 121 or by the administrative program 127. The message queuing program 142 itself is unaware of the pub / sub program 129 or its intentions.

[0063] In the examples herein, the relationship between queues and workers does not have to be one-to-one. For example, there may be a one-to-N queue-to-worker relationship for each queue 145, where N is an integer greater than 0. Thus, each message queue 145 may have at least one PE worker instance 212 for polling and consuming events, but some message queues 145 may have multiple PE worker instances 212 for polling and consuming events.

[0064] In addition, each PE worker instance 212 may include a self-management algorithm 141 for determining its own operation, and may also access the self-management algorithm 141. For example, the management program 127 may collect or receive system configuration information indicating the current state of each queue 145. For example, as indicated by the arrow in 216, the management program 127 may communicate with the message queuing program 142 to receive the current status of each message queue 145, such as how full each queue 145 is, how many of the queues 145 are allocated to specific types of designations, and may provide this information to the pub / sub program 129. Similarly, the management program 127 may receive information related to the PE worker instance 212 from the policy engine program 140.

[0065] The management program 127 may continuously provide this system configuration information to the pub / sub program 129 as policy engine configuration information 218, which can then be made available to the message queuing program 142 as a PE configuration stream 220 for the policy engine worker instance 212. As will be further discussed below, the PE worker instance 212 may receive the PE configuration information 218 through the PE configuration stream 220, such as based on update notifications, and may execute a self-management algorithm 141 to self-determine which queue 145 to consume events from based on the received PE configuration information 218.

[0066] The PE configuration stream 220 provides a mechanism for the system 200 to distribute work across the entire set of queues 145, enabling scaling of PE worker instances 212 to ensure that each message queue 145 is served and that the PE worker instances 212 distribute themselves fairly and evenly across the entire set of queues 145. In the examples herein, the management program 127 and / or management user may add / remove message queues 145 and / or PE worker instances 212 as discussed above to achieve optimal system operation while ensuring that all message queues 145 are served even if the system topology changes.

[0067] In the examples herein, there is no “assignment” of PE worker instances 212 to a specific queue. Rather, in the examples herein, the queuing framework 144 holds submitted work and provides that work to PE worker instances 212. When the work becomes available, PE worker instances 212 pull and consume the work. As discussed above, the message queuing program 142 may be instructed, for example, via one or more API calls, to add or remove queues 145 based on workload and availability. Furthermore, as discussed above, PE worker instances 212 may also be added or removed via one or more API calls based on load and availability. The self-managing algorithm 141 may operate to cause PE worker instances 212 to distribute the PE worker instances 212 across the message queues in an optimal manner in order to balance the load on each PE worker instance 212 so that it is as equal as possible, and at the same time ensure that no message queue 145 is orphaned or abandoned. For example, each PE worker instance 212 may decide which queue 145 to consume from, without receiving an allocation from a central authority, based on a self-managing algorithm 141 that can take into account the current system topology.

[0068] Thus, in the example herein, the message queue 145 and the PE worker instance 212 are decoupled. In particular, the PE worker instance 212 determines which queue it should bind to, based on the type of work it can perform. Furthermore, a client may publish to the message queue 145 regarding the work it needs to perform, but beyond that, there is no affinity or tracking, and no centralized component is needed to select a specific worker for a specific queue. Rather, in the system herein, queues and workers are self-managing.

[0069] Figure 3 is a sequence diagram illustrating an exemplary process 300 in which, at startup for several implementations, a PE worker instance maps itself to a queue or removes itself from a queue. In the examples herein, the pub / sub program 129 can notify all participants of changes in the queue count and / or worker count so that each PE worker instance 212 can decide whether to reconfigure its respective queue mapping.

[0070] Each PE worker instance 212 may maintain its own persistent identifier (ID) that is unique within the system, such as a "UUID" or any other individually distinguishable ID. Furthermore, at startup, each PE worker instance 212 may perform an initialization step, which may include registering each PE worker instance 212 itself with the pub / sub program 129 to announce that each PE worker instance 212 is available and ready to work, subscribing to receive change notifications from the pub / sub program 129, reading the current configuration, and determining its queue mapping.

[0071] In 302, at each startup, the PE worker instance 212 may idempotently register its UUID with the common KEY / worker of the pub / sub program 129, thereby providing notification of the configuration change to other existing PE worker instances 212.

[0072] In 304, PE worker instances 212 may subscribe to a pub / sub program 129 for changes to a common key / configuration. For example, in one embodiment, the pub / sub program 129 may have a client (e.g., PE worker instance 212) set the value of a key and later retrieve that value as a map-type interface. Thus, a client can request notification for key "X", and if "X" is updated, the publication mechanism of the pub / sub program 129 can send an update to the listening clients (each PE worker instance 212) indicating that the value of this key has changed and that the client should run its self-managing algorithm to determine if the change is necessary. Thus, by subscribing to the pub / sub program 129, PE worker instances 212 are notified of scaling events of other PE worker instances 212 (e.g., increases or decreases in the PE worker instance count), and as a result, individual PE worker instances can recalculate their queue allocations.

[0073] In 306, PE worker instance 212 may request (obtain) existing configuration information.

[0074] In 308, the PE worker instance 212 may receive the requested existing configuration from the pub / sub program 129, for example, via the PE configuration information stream 220, as discussed above with respect to Figure 2.

[0075] In step 310, the PE worker instance 212 may submit configuration information to the self-management algorithm 141.

[0076] In 312, the PE worker instance 212 receives a mapping to the designated queue from the self-managing algorithm 141.

[0077] In 314, the PE worker instance 212 applies the received mapping, for example, by mapping to the designated message queue 145 and / or by unmapping to the current message queue 145. As mentioned above, for example, the PE worker instance 212 may send an API call to the message queuing program 142 to bind to the selected queue or to unbind from a previously bound queue.

[0078] Figure 4 is a sequence diagram illustrating an exemplary process 400 in which PE worker instances map to different queues and / or remove themselves from queues during runtime in several implementation forms. For example, during runtime, PE worker instance 212 invokes a handler that reads the system's current configuration as a subscriber to a pub / sub program 129 that PE worker instance 212 registered during initialization. A self-management algorithm 141 is used to recalculate the appropriate queue mapping for PE worker instance 212 and adjust queue consumption as needed.

[0079] As an example, suppose a PE worker instance 212 currently assigned to queue X runs its self-management algorithm 141 based on the most recently received configuration information, and the self-management algorithm 141 decides to keep queue X as the queue to which this PE worker instance 212 is mapped. In this situation, no further action is required. On the other hand, suppose a PE worker instance 212 currently assigned to queue X runs its self-management algorithm 141, and the self-management algorithm 141 indicates that the PE worker instance 212 should be mapped to queue Y instead of queue X. Thus, the PE worker instance 212 may unbind itself from queue X, perform the necessary cleanup with respect to queue X, and then bind to queue Y and begin consuming from queue Y.

[0080] In 402, PE worker instance 212 may request (obtain) existing system configuration information.

[0081] In case 404, the PE worker instance 212 may receive the requested existing system configuration information from the pub / sub program 129.

[0082] In 406, the PE worker instance 212 may submit configuration information to the self-management algorithm 141.

[0083] In step 408, the PE worker instance 212 receives a mapping to the designated queue from the self-managing algorithm 141.

[0084] In 410, the PE worker instance 212 applies the received mapping, for example, by mapping to the designated message queue 145 and / or by unmapping to the current message queue 145.

[0085] Figure 5 shows an exemplary data structure 500 used by a self-managing algorithm 141 in several implementation forms. In this example, the data structure 500 includes an index 502, a queue count 504, and a result 506.

[0086] For example, the self-management algorithm 141 may receive the UUID of the PE worker instance 212 making the mapping request, the type of queue the node is targeting, a consistently ordered list of PE worker instances 212 (e.g., lexicographically ordered by UUID) and a list according to the type in the system, the current count of queues (by type) in the system, and a consistent, well-defined naming pattern for the queues. In some examples, the PE worker instance 212 may have a fixed type similar to the type of each queue. In other examples, the PE worker instance 212 may have multiple types and / or perform multiple different types of work and / or draw work from different types of queues.

[0087] For example, the queue naming pattern may be "type.count". For example, suppose there are currently six queues in the system, two of which are of type 1 "processing-A", three of which are of type 2 "processing-B", and one of which is of type 3 "processing-C". Based on the queue naming pattern type.count, these six queues may be named processing-A.1, processing-A.2, processing-B.1, processing-B.2, processing-B.3, and processing-C.1.

[0088] In some examples, the management program 127 may be responsible for maintaining a continuous one-to-many naming pattern for a particular type of queue without holes. The self-managing algorithm 141 may receive the UUID of the requesting PE worker instance 212 and a list of all PE worker instances 212. If the self-managing algorithm 141 finds the UUID of the requesting PE worker instance in the list, the algorithm may use index 502 to determine the queue mapping using modulo division of index % queue count to determine the result 506. For example, the algorithm performing the calculation has a list of PE worker instances 212 and a list of queues (by type) that currently exist in the system. In the list of PE worker instances 212, there exists a PE worker instance 212 "itself" that executes the self-managing algorithm 141. The algorithm orders the list to find the offset of where the PE worker instance 212 "itself" is located in the list. Thus, in this example, we assume that there are currently three PE worker instances 212 in the system, namely instance A, instance B, and instance C. In this case, instance "B" corresponds here to index "1". Based on applying this index value to data structure 500, in the index (502) column, the second row corresponds to the index value "1". The queue count is provided by the second area of ​​the configuration state maintained by the pub / sub program. Thus, based on the desired type, the algorithm may determine the queue count and populate its value as the queue count 504. The modulo value obtained by dividing the index value by the queue count value is the queue allocation value 506. The algorithm may then use the queue naming pattern described to identify the queue binding target based on the type and result 506. Using result 506 and the target type, the mapping algorithm constructs the expected queue name, for example, "processing-B.3" in this example, to return the queue for consumption by a particular PE worker instance 212.

[0089] This algorithm may rely on the system (e.g., management program 127) to ensure that the queue count for a particular queue type never exceeds the worker count; otherwise, it should be noted that queues may become orphaned. Therefore, a PE worker instance 212 may be responsible for listening to only one queue per type. Otherwise, a more complex mapping algorithm may be employed to ensure that workers are assigned to all queues, in which case workers may be responsible for more than one queue per type. This design requires configuration management to ensure that there are one-to-one or one-to-many queue-to-worker relationships, and that a particular PE worker instance 212 is responsible for serving a single queue type.

[0090] Figure 6 shows an example of a data structure 600 for state tracking in several implementation configurations. In this example, the data structure 600 includes a key 602 and a value 604. For example, the key 602 is KEY / configuration, and the value 604 is a JSON blob or any other data format representing the current system configuration, and may include information such as the current queue count or a list of PE worker instances 212 currently present in the system. In some examples, the pub / sub program 129 may be configured to track the state of queues and PE worker instances 212 so that it may provide persistence, redundancy, and notification.

[0091] Key 602 should be readable by all PE worker instances 212, and only the management program 127 should write value 604. All PE worker instances 212 register a watcher / listener with the pub / sub program 129. When value 604 changes, each PE worker instance 212 may recalculate their mappings based on the latest configuration represented by value 604. This simplifies worker logic and also simplifies debugging by subscribing to only a single source, readable from a single location, for receiving current system state information.

[0092] Figure 7 shows exemplary pseudocode 700 illustrating an example of the current system state for several implementation configurations. For example, in Figure 7, as shown in 702, there are three types of queues, namely "processing-A", "processing-B", and "processing-C", and the current count for each type is shown following the type. In addition, as shown in 704, "engines" shows the UUID of the PE worker instance 212 currently associated with each queue.

[0093] Figure 8 shows an exemplary data structure 800 that can be used by a pub / sub program 129 in several implementation forms to track PE worker instances. In this example, the data structure 800 includes a key 802 and a value 804. The key 802 may be readable by all PE worker instances 212. Furthermore, only PE worker instances 212 should write under the parent key 806 to announce addition to or removal from the system. As mentioned above, for example with respect to Figure 3, when a PE worker instance 212 is initialized, if there is no entry in the data structure 800, the PE worker instance 212 adds itself to the data structure 800 as a child node. When a PE worker instance 212 is permanently decommissioned, the PE worker instance 212 may remove its associated child node entry from the data structure 800.

[0094] As an example, the management program 127 may register a recursive and persistent listener with the pub / sub program 129. For example, the listener (management program 127) registers a subscription with the pub / sub program 129 to be notified of events published to subscribers by the pub / sub program 129. The management program 127 may also react to any additions or deletions of PE worker instances 212 by regenerating the current configuration JSON and updating it if the configuration node values ​​have changed, thereby signaling the PE worker instances 212 that their mappings should be recalculated. This operation should output a consistent and ordered list of workers, so that each worker can index itself in the list in a consistent manner without reordering. If a PE worker instance 212 cannot find itself under column 802 in the list, it must wait and / or retry, assuming that the management task is still calculating the state or that the updated state has not yet refreshed the cached values.

[0095] Figure 9 shows an exemplary data structure 900 that illustrates the current queue type count for several implementation configurations. In this example, the data structure 900 includes key 902 and value 904. In the data structure 900, "current" represents an active configuration where supply is being made to the queue portion of the configuration node and it can always exist. Thus, the data structure 900 provides the system's queue allocation. As previously mentioned, this may be determined by the administrative user and / or via the administrative program 127. In some examples, the administrative program 127 may include an automated process for monitoring and determining the scale up / down of the queue. Thus, the data structure 900 may be updated in response to changes in the queue count.

[0096] Figure 10 shows exemplary components of a service computing unit 102 that may be used to implement at least some of the functions of the system described herein. The service computing unit 102 may include one or more servers or other types of computing devices that can be embodied in various ways. For example, in the case of a server, programs, other functional components, and data may be implemented in a single server, a cluster of servers, a server farm or data center, a cloud-hosted computing service, etc., but other computer architectures may be used in addition to or instead thereof. Multiple service computing units 102 may be located together or separately, and may be organized, for example, as virtual servers, server banks, and / or server farms. The functions described may be provided by servers of a single entity or enterprise, or by servers and / or services of multiple different entities or enterprises.

[0097] In the example shown, the service arithmetic unit 102 may include, or be associated with, one or more processors 1002, one or more computer-readable media 1004, and one or more communication interfaces 1006. Each processor 1002 may be a single processing unit or multiple processing units, and may include one or more arithmetic units or multiple processing cores. The processor 1002 may be implemented as one or more central processing units, microprocessors, microcomputers, microcontrollers, digital signal processors, state machines, logic circuits, and / or any device that manipulates signals based on operation instructions. As an example, the processor 1002 may include one or more hardware processors and / or logic circuits of any suitable type that are specifically programmed or configured to perform the algorithms and processes described herein. The processor 1002 may be configured to fetch and execute computer-readable instructions stored in the computer-readable media 1004, which can be programmed to perform the functions described herein.

[0098] The computer-readable medium 1004 may include volatile and non-volatile memory and / or removable and non-removable media implemented in any type of technology for storing information such as computer-readable instructions, data structures, program modules, or other data. For example, the computer-readable medium 1004 may include, but is not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, optical storage, solid-state storage, magnetic tape, magnetic disk storage, storage arrays, network-attached storage, storage area networks, cloud storage, or any other media that can be used to store desired information and can be accessed by computing devices. Depending on the configuration of the service computing unit 102, the computer-readable medium 1004 may be a tangible non-temporary medium, only if it is mentioned that non-temporary computer-readable media exclude media such as energy, carrier signals, electromagnetic waves, and / or signals themselves. In some cases, the computer-readable medium 1004 may be located in the same place as the service computing unit 102, and in other cases, the computer-readable medium 1004 may be partially separated from the service computing unit 102.

[0099] The computer-readable medium 1004 may be used to store any number of functional components that can be executed by the processor 1002. In many implementations, these functional components include instructions or programs that are executable by the processor 1002 and, when executed, specifically program the processor 1002 to perform operations belonging to the service arithmetic unit 102 as herein. The functional components stored in the computer-readable medium 1004 may include a user web application 116, a management web application 124, a management program 127, an API 128, a pub / sub program 129, a storage program 130, a metadata gateway program 138, a policy engine program 140 including a self-management algorithm 141, and a message queuing program 142, each of which may include one or more computer programs, applications, executable code, or parts thereof. Furthermore, although these programs are shown together in this example, some or all of these programs may be executed on a separate service arithmetic unit 102 during use.

[0100] In addition, the computer-readable medium 1004 may store data, data structures, and other information used to perform the functions and services described herein. For example, the computer-readable medium 1004 may store a metadata database 134 and local object data 137. Furthermore, although these data structures are shown together in this example, some or all of these data structures may be stored in a separate service arithmetic unit 102 during use. The service arithmetic unit 102 may also include or maintain other functional components and data, which may include programs, drivers, etc., as well as other data used or generated by the functional components. Furthermore, the service arithmetic unit 102 may include many other logical components, program components, and physical components, the above-mentioned ones being merely examples relevant to the discussion herein.

[0101] One or more communication interfaces 1006 may include one or more software and hardware components to enable communication with various other devices, such as on one or more networks 106 and 107. For example, communication interface 1006 may enable communication via one or more of the following: LAN, Internet, cable network, cellular network, wireless network (e.g., Wi-Fi) and wired network (e.g., Fibre Channel, optical fiber, Ethernet), direct connection, Bluetooth® and other short-range communications, and similar ones, as further enumerated elsewhere in this specification.

[0102] The exemplary processes described herein are merely examples of processes provided for discussion. Many other variations will be apparent to those skilled in the art based on the disclosure herein. Furthermore, while the disclosure herein provides several examples of preferred frameworks, architectures, and environments for performing the processes, the implementations herein are not limited to the specific examples shown and discussed. Moreover, this disclosure provides a variety of exemplary implementations, as described and shown in the drawings. However, this disclosure is not limited to the implementations described and shown herein and may extend to other implementations known or to be known to those skilled in the art.

[0103] The various instructions, methods, and techniques described herein may be considered in the general context of computer executable instructions, such as computer programs and applications stored in computer-readable media and executed by a processor. Generally, the terms "program" and "application" may be used interchangeably and may include instructions, routines, modules, objects, components, data structures, executable code, etc., for performing a particular task or implementing a particular data type. These programs, applications, and similar entities may be executed as native code, or they may be downloaded and executed in a virtual machine or other just-in-time compilation execution environment. Typically, the functionality of programs and applications may be combined or distributed as desired in various implementations. Implementations of these programs, applications, and techniques may be stored in computer storage media or transmitted via some form of communication medium.

[0104] While the subject matter is described using language specific to structural features and / or methodological actions, it should be understood that the subject matter as defined in the attached claims is not necessarily limited to the specific features or actions described. Rather, specific features and actions are disclosed as exemplary forms that implement the claims.

Claims

1. A system comprising one or more processors configured to perform operations by executable instructions, wherein the operations are: Creating multiple queues configured to receive data, The process involves generating multiple worker instances, each worker instance being configured to be associated with each of the multiple queues in order to consume data from that queue. The individual worker instances of the plurality of worker instances are configured to perform an action, and the action is, Receiving system configuration information indicating the state of each of the aforementioned queues and the state of each of the aforementioned worker instances, Each of the individual worker instances determines, at least in part, based on the received system configuration information, whether to change the association of the individual worker instance from the respective queue to which it is currently associated to a different queue of the plurality of queues. Includes, A system in which, based on at least the received system configuration information, at least one of the individual worker instances decides to change its association from consuming data from a first queue of the queues to consuming data from a second queue of the queues.

2. The system according to claim 1, wherein the system configuration information further includes an instruction for each type of queue and a count of the number of queues of each type in the system.

3. The system according to claim 2, wherein each of the aforementioned types of each queue is based at least in part on the characteristics of the data received by each of the aforementioned queues.

4. The system according to claim 2, wherein the system configuration information further includes instructions on which worker instance is associated with which queue among the respective queues.

5. The system according to claim 1, wherein, following the creation of each worker instance, each worker instance registers itself in order to receive the system configuration information in response to changes in the system configuration.

6. The system according to claim 1, wherein the data received by the queue and consumed by the worker instance corresponds to a plurality of events, and at least some of the respective events correspond to data operations including at least one of a write request, a delete request, and a move request.

7. The aforementioned operation is, Receiving metrics related to queue and worker performance, Based at least on the received metrics, a decision is made to add a queue to the plurality of queues, Send an Application Programming Interface (API) call to a message queuing program running on one or more of the aforementioned processors, causing the message queuing program to generate a new queue. The system according to claim 1, further comprising:

8. The system according to claim 1, wherein the plurality of worker instances are configured to receive the system configuration information from a publication / subscription program that maintains the system configuration information.

9. The system according to claim 1, wherein the worker instances are configured to ensure that each queue is associated with at least one worker instance.

10. The system according to claim 1, wherein the operation further includes each worker instance transmitting the data consumed from each queue to its respective target recipient.

11. The system according to claim 1, wherein at least one of the individual worker instances that decides to change its association from consuming data from the first queue of the queues to consuming data from the second queue of the queues sends a first application programming interface (API) call to unbind from the first queue and a second API call to bind to the second queue.

12. One or more processors generate multiple queues configured to receive data, The process involves generating multiple worker instances using one or more processors, wherein each worker instance is configured to be associated with each of the multiple queues in order to consume data from that queue. A method comprising the following, wherein each worker instance of the plurality of worker instances is configured to perform an action, and the action is Receiving system configuration information indicating the state of each of the aforementioned queues and the state of each of the aforementioned worker instances, Each of the individual worker instances determines, at least in part, based on the received system configuration information, whether to change the association of the individual worker instance from the respective queue to which it is currently associated to a different queue of the plurality of queues. Includes, A method for determining, based at least on the received system configuration information, that at least one of the individual worker instances changes its association from consuming data from a first queue of the queues to consuming data from a second queue of the queues.

13. The method according to claim 12, wherein the system configuration information further includes an instruction for each type of queue and a count of the number of queues of each type in the system.

14. A non-temporary computer-readable medium for storing instructions, wherein, when an instruction is executed by one or more processors, the one or more processors are configured to perform an operation, and the operation is: Creating multiple queues configured to receive data, The process involves generating multiple worker instances, each worker instance being configured to be associated with each of the multiple queues in order to consume data from that queue. The individual worker instances of the plurality of worker instances are configured to perform an action, and the action is, Receiving system configuration information indicating the state of each of the aforementioned queues and the state of each of the aforementioned worker instances, Each of the individual worker instances determines, at least in part, based on the received system configuration information, whether to change the association of the individual worker instance from the respective queue to which it is currently associated to a different queue of the plurality of queues. Includes, A non-temporary computer-readable medium in which, based at least on the received system configuration information, at least one of the individual worker instances decides to change its association from consuming data from a first queue of the queues to consuming data from a second queue of the queues.

15. The non-temporary computer-readable medium according to claim 14, wherein the system configuration information further includes an instruction for each type of queue and a count of the number of such queues of each type in the system.

Images