Methods, systems, and computer program products for dynamic data allocation to nodes in a distributed system

Abstract

A method, system, and computer program product are provided for dynamically allocating data to nodes in a distributed system. The method can include storing a data block at a node; receiving node health status data associated with the node from the node; and, for the data block, performing dynamic data allocation, the dynamic data allocation including: receiving existence time data associated with the existence time of the data block from the node on which the data block is stored; receiving network latency data associated with the network latency used by the client to access the data block from each client accessing the data block at the node; generating a node assignment for the data block based on (i) the node health status data, (ii) the existence time data, and (iii) the network latency data; and moving the data block from the current node to the node assignment in response to determining that the node assignment is different from the current node.

Description

Technical Field

[0001] This disclosure relates in its entirety to distributed computing, and in non-limiting embodiments or aspects to methods, systems, and computer program products for dynamically distributing data to nodes in a distributed system. Background Technology

[0002] Improving the utilization of clusters or nodes is a common challenge in distributed computing. Several factors can lead to suboptimal utilization of clusters or nodes, such as how data is stored on the cluster or nodes, and the cost of retrieving data from the cluster or nodes. For example, data may be stored on one or more data nodes located far from where clients attempting to access the data are located, or data may be stored on hardware systems of data nodes that are unable to operate at full capacity for some reason.

[0003] The problem of efficient data allocation on a distributed cluster is not a static one, nor can it be solved by creating a specific solution. A more dynamic approach is needed to ensure that the needs of clients attempting to access data are met, and that the data storage solution evolves as client requirements change. Furthermore, the problem of efficient data storage is not limited to any specific data storage format (such as file storage, object storage, etc.), but spans a range of data storage formats. Therefore, a dynamic data allocation solution that is not limited to specific use cases is required. Summary of the Invention

[0004] Therefore, improved methods, systems, and computer program products are provided for dynamic data allocation to nodes in a distributed system.

[0005] According to a non-limiting embodiment or aspect, a method is provided, comprising: storing multiple data blocks at multiple nodes in a distributed system using at least one processor; receiving node health status data associated with multiple read latency associated with the multiple nodes using the at least one processor; and performing dynamic data allocation for data blocks among the multiple data blocks using the at least one processor, the dynamic data allocation comprising: receiving existence time data associated with the existence time (age) of the data block in the distributed system from one or more nodes of the multiple nodes on which the data block was stored in a previous time period; and receiving data from a number of clients who accessed the data block at one or more nodes in a previous time period. Each client in the terminal receives network latency data associated with the network latency used by that client to access the data block at one or more nodes; based on (i) node health status data, (ii) time-of-existence data and (iii) network latency data for each of the number of clients accessing the data block at one or more nodes, a node assignment for the data block is generated among a plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0006] In some non-limiting embodiments or aspects, for each of the multiple nodes, node health data associated with multiple read latency related to the multiple nodes includes read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0007] In some non-limiting embodiments or aspects, the node health status data also includes multiple load factors associated with multiple nodes, and wherein for a data block in a plurality of data blocks: the network latency data also includes the level priority of each of the number of clients accessing the data block at one or more nodes, and the node assignment of the node for the data block among the plurality of nodes is further based on the multiple load factors associated with the multiple nodes and the level priority of each of the number of clients accessing the data block at one or more nodes.

[0008] In some non-limiting embodiments or aspects, where for a data block among multiple data blocks, the existence time data associated with the existence time of the data block includes the number of clients that accessed the data block at one or more nodes in a previous time period and the total number of times the data block was accessed at one or more nodes in a previous time period.

[0009] In some non-limiting embodiments or aspects, the method further includes, for a data block among a plurality of data blocks, utilizing the at least one processor, for each of the number of clients that accessed the data block at one or more nodes in a previous time period, determining the network latency for the client to access the data block at one or more nodes based on network latency data associated with the network latency for the client to access the data block at one or more nodes and the read latency of one or more nodes.

[0010] In some non-limiting embodiments or aspects, generating a node assignment for a data block among multiple data blocks includes: providing the following as input to a machine learning model: (i) multiple read latencies associated with the multiple nodes, (ii) the number of clients accessing the data block at one or more nodes in a previous time period, (iii) the total number of times the data block was accessed at one or more nodes in a previous time period, and (iv) for each of the clients accessing the data block at one or more nodes, the network latency of the client used to access the data block at one or more nodes; and receiving the node assignment for the data block among the multiple nodes as output from the machine learning model.

[0011] In some non-limiting embodiments or aspects, the method further includes: for each of a plurality of data blocks, using the at least one processor to perform dynamic data allocation continuously or periodically.

[0012] According to some non-limiting embodiments or aspects, a system is provided, comprising: at least one processor coupled to a memory and configured to: store a plurality of data blocks at a plurality of nodes in a distributed system; receive node health status data associated with a plurality of read latency associated with the plurality of nodes from the plurality of nodes; and perform dynamic data allocation for data blocks among the plurality of data blocks, the dynamic data allocation comprising: receiving existence time data associated with the existence time of the data block in the distributed system from one or more nodes of the plurality of nodes on which the data block was stored in a previous time period; and receiving data from a number of clients that accessed the data block at one or more nodes in a previous time period. Each client in the process receives network latency data associated with the network latency used by that client to access the data block at one or more nodes; based on (i) node health status data, (ii) time-of-existence data and (iii) network latency data for each of the number of clients accessing the data block at one or more nodes, a node assignment for the data block is generated among a plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0013] In some non-limiting embodiments or aspects, for each of the multiple nodes, node health data associated with multiple read latency related to the multiple nodes includes read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0014] In some non-limiting embodiments or aspects, the node health status data also includes multiple load factors associated with multiple nodes, and wherein for a data block in a plurality of data blocks: the network latency data also includes the level priority of each of the number of clients accessing the data block at one or more nodes, and the node assignment of the node for the data block among the plurality of nodes is further based on the multiple load factors associated with the multiple nodes and the level priority of each of the number of clients accessing the data block at one or more nodes.

[0015] In some non-limiting embodiments or aspects, for a data block among multiple data blocks, the existence time data associated with the existence time of the data block includes the number of clients that accessed the data block at one or more nodes in a previous time period and the total number of times the data block was accessed at one or more nodes in a previous time period.

[0016] In some non-limiting embodiments or aspects, the at least one processor is further configured to: for a data block among a plurality of data blocks: for each of the number of clients that accessed the data block at one or more nodes in a previous time period, determine the network latency for the client to access the data block at one or more nodes based on network latency data associated with the network latency for the client to access the data block at one or more nodes and the read latency of one or more nodes.

[0017] In some non-limiting embodiments or aspects, generating a node assignment for a data block among multiple data blocks includes: providing the following as input to a machine learning model: (i) multiple read latencies associated with the multiple nodes, (ii) the number of clients accessing the data block at one or more nodes in a previous time period, (iii) the total number of times the data block was accessed at one or more nodes in a previous time period, and (iv) for each of the clients accessing the data block at one or more nodes, the network latency of the client used to access the data block at one or more nodes; and receiving the node assignment for the data block among the multiple nodes as output from the machine learning model.

[0018] In some non-limiting embodiments or aspects, the at least one processor is further configured to perform dynamic data allocation continuously or periodically for each of a plurality of data blocks.

[0019] According to some non-limiting embodiments or aspects, a computer program product is provided, comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium including program instructions that, when executed by at least one processor of a plurality of nodes, cause the at least one processor to: store a plurality of data blocks at the plurality of nodes in a distributed system; receive node health status data associated with a plurality of read delays associated with the plurality of nodes from the plurality of nodes; and perform dynamic data allocation for the data blocks among the plurality of data blocks, the dynamic data allocation comprising: receiving existence time data associated with the existence time of the data block in the distributed system from one or more of the plurality of nodes on which the data block was stored in a previous time period; ... a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of the data block in a previous time period; receiving existence time data associated with the existence time of Each of a number of clients accessing the data block at one or more nodes receives network latency data associated with the network latency for that client to access the data block at one or more nodes; based on (i) node health status data, (ii) time-of-existence data and (iii) network latency data for each of the number of clients accessing the data block at one or more nodes, a node assignment for the data block is generated among a plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0020] In some non-limiting embodiments or aspects, for each of the multiple nodes, node health data associated with multiple read latency related to the multiple nodes includes read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0021] In some non-limiting embodiments or aspects, the node health status data also includes multiple load factors associated with multiple nodes, and wherein for a data block in a plurality of data blocks: the network latency data also includes the level priority of each of the number of clients accessing the data block at one or more nodes, and the node assignment of the node for the data block among the plurality of nodes is further based on the multiple load factors associated with the multiple nodes and the level priority of each of the number of clients accessing the data block at one or more nodes.

[0022] In some non-limiting embodiments or aspects, for a data block among a plurality of data blocks, the existence time data associated with the existence time of the data block includes the number of clients accessing the data block at one or more nodes in a previous time period and the total number of times the data block was accessed at one or more nodes in the previous time period, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for a data block among a plurality of data blocks: for each of the number of clients accessing the data block at one or more nodes in the previous time period, determine the network latency for the client to access the data block at one or more nodes based on network latency data associated with the network latency for the client to access the data block at one or more nodes and the read latency of one or more nodes.

[0023] In some non-limiting embodiments or aspects, generating a node assignment for a data block among multiple data blocks includes: providing the following as input to a machine learning model: (i) multiple read latencies associated with the multiple nodes, (ii) the number of clients accessing the data block at one or more nodes in a previous time period, (iii) the total number of times the data block was accessed at one or more nodes in a previous time period, and (iv) for each of the clients accessing the data block at one or more nodes, the network latency of the client used to access the data block at one or more nodes; and receiving the node assignment for the data block among the multiple nodes as output from the machine learning model.

[0024] In some non-limiting embodiments or aspects, the program instructions, when executed by the at least one processor, further cause the at least one processor to: perform dynamic data allocation continuously or periodically for each of the plurality of data blocks.

[0025] Other non-restrictive implementation schemes or aspects are set forth in the following numbered clauses:

[0026] Clause 1. A method comprising: storing a plurality of data blocks at a plurality of nodes in a distributed system using at least one processor; receiving, using the at least one processor, node health status data associated with a plurality of read latency associated with the plurality of nodes; and, for a data block among the plurality of data blocks, performing dynamic data allocation using the at least one processor, the dynamic data allocation comprising: receiving, from one or more of the plurality of nodes on which the data block was stored in a previous time period, existence time data associated with the existence time of the data block in the distributed system; and receiving, from each of a plurality of clients that accessed the data block at the one or more nodes in the previous time period, data related to the client's access to the data block in the distributed system. The network latency data associated with the network latency of accessing the data block at one or more nodes; based on (i) the node health status data, (ii) the existence time data and (iii) the network latency data for each of the number of clients accessing the data block at the one or more nodes, a node assignment for the data block is generated among the plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0027] Clause 2. The method as described in Clause 1, wherein for each of the plurality of nodes, the node health status data associated with the plurality of read latency associated with the plurality of nodes includes the read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0028] Clause 3. The method as described in Clause 1 or Clause 2, wherein the node health status data further includes a plurality of load factors associated with the plurality of nodes, and wherein for the data block among the plurality of data blocks: the network latency data further includes a level priority of each of the number of clients accessing the data block at the one or more nodes, and the node assignment of the node for the data block among the plurality of nodes is generated further based on the plurality of load factors associated with the plurality of nodes and the level priority of each of the number of clients accessing the data block at the one or more nodes.

[0029] Clause 4. The method of any one of Clauses 1 to 3, wherein for the data block among the plurality of data blocks, the existence time data associated with the existence time of the data block includes the number of clients accessing the data block at the one or more nodes during the previous time period and the total number of times the data block was accessed at the one or more nodes during the previous time period.

[0030] Clause 5. The method of any one of Clauses 1 to 4 further comprises: for the data block among the plurality of data blocks, utilizing the at least one processor: for each of the number of clients that accessed the data block at the one or more nodes during the previous time period, determining the network latency for the client to access the data block at the one or more nodes based on network latency data associated with the network latency for the client to access the data block at the one or more nodes and the read latency of the one or more nodes.

[0031] Clause 6. The method of any one of Clauses 1 to 5, wherein generating the node assignment of the node for the data block among the plurality of data blocks comprises: providing the following as input to a machine learning model: (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients accessing the data block at one or more nodes in the previous time period, (iii) the total number of times the data block was accessed at one or more nodes in the previous time period, and (iv) the network latency for each of the number of clients accessing the data block at one or more nodes; and receiving the node assignment of the node for the data block among the plurality of nodes as output from the machine learning model.

[0032] Clause 7. The method of any one of Clauses 1 to 6 further comprises: for each of the plurality of data blocks, performing the dynamic data allocation continuously or periodically using the at least one processor.

[0033] Clause 8. A system comprising: at least one processor coupled to a memory and configured to: store a plurality of data blocks at a plurality of nodes in a distributed system; receive node health status data associated with a plurality of read latency associated with the plurality of nodes from the plurality of nodes; and perform dynamic data allocation for data blocks among the plurality of data blocks, the dynamic data allocation comprising: receiving, from one or more of the plurality of nodes on which the data blocks were stored in a previous time period, existence time data associated with the existence time of the data blocks in the distributed system; and receiving, from each of a plurality of clients that accessed the data blocks at the one or more nodes in the previous time period, data with the client's access to the data blocks in the distributed system. The network latency data associated with the network latency of accessing the data block at one or more nodes; based on (i) the node health status data, (ii) the existence time data and (iii) the network latency data for each of the number of clients accessing the data block at the one or more nodes, a node assignment for the data block is generated among the plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0034] Clause 9. The system as described in Clause 8, wherein for each of the plurality of nodes, the node health status data associated with the plurality of read latency associated with the plurality of nodes includes the read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0035] Clause 10. A system as described in Clause 8 or Clause 9, wherein the node health status data further includes a plurality of load factors associated with the plurality of nodes, and wherein for the data block among the plurality of data blocks: the network latency data further includes a level priority of each of the number of clients accessing the data block at the one or more nodes, and the node assignment of the node for the data block among the plurality of nodes is generated further based on the plurality of load factors associated with the plurality of nodes and the level priority of each of the number of clients accessing the data block at the one or more nodes.

[0036] Clause 11. The system of any one of Clauses 8 to 10, wherein for the data block among the plurality of data blocks, the existence time data associated with the existence time of the data block includes the number of clients accessing the data block at the one or more nodes during the previous time period and the total number of times the data block was accessed at the one or more nodes during the previous time period.

[0037] Clause 12. The system of any one of Clauses 8 to 11, wherein the at least one processor is further configured to: for each of the plurality of data blocks: for each of the number of clients that accessed the data block at the one or more nodes during the previous time period, determine the network latency of the client for accessing the data block at the one or more nodes based on the network latency associated with the network latency of the client for accessing the data block at the one or more nodes and the read latency of the one or more nodes.

[0038] Clause 13. The system of any one of Clauses 8 to 12, wherein generating the node assignment of the node for the data block among the plurality of data blocks comprises: providing the following as input to a machine learning model: (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients accessing the data block at one or more nodes in the previous time period, (iii) the total number of times the data block was accessed at one or more nodes in the previous time period, and (iv) the network latency for each of the number of clients accessing the data block at one or more nodes; and receiving the node assignment of the node for the data block among the plurality of nodes as output from the machine learning model.

[0039] Clause 14. The system of any one of Clauses 8 to 13, wherein the at least one processor is further configured to perform the dynamic data allocation continuously or periodically for each of the plurality of data blocks.

[0040] Clause 15. A computer program product comprising at least one non-transitory computer-readable medium, the at least one non-transitory computer-readable medium comprising program instructions, which, when executed by at least one processor of a plurality of nodes, cause the at least one processor to: store a plurality of data blocks at a plurality of nodes in a distributed system; receive from the plurality of nodes node health status data associated with a plurality of read latency associated with the plurality of nodes; and perform dynamic data allocation for a data block among the plurality of data blocks, the dynamic data allocation comprising: receiving from one or more of the plurality of nodes on which the data block was stored in a previous time period existence time data associated with the existence time of the data block in the distributed system existence time data; and receiving from a certain number of nodes that accessed the data block at the one or more nodes in the previous time period existence time data. Each of the number of clients receives network latency data associated with the network latency of the client for accessing the data block at the one or more nodes; based on (i) the node health status data, (ii) the time-of-existence data and (iii) the network latency data for each of the number of clients accessing the data block at the one or more nodes, a node assignment for the data block is generated among the plurality of nodes; it is determined whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, the data block is moved from the current node to the node assigned to the node.

[0041] Clause 16. The computer program product as described in Clause 15, wherein for each of the plurality of nodes, the node health status data associated with the plurality of read latency associated with the plurality of nodes includes the read latency, which includes the average amount of time consumed by the node to serve one or more requests for access to one or more data blocks at the node.

[0042] Clause 17. A computer program product as described in Clause 15 or Clause 16, wherein the node health status data further includes a plurality of load factors associated with the plurality of nodes, and wherein for the data block among the plurality of data blocks: the network latency data further includes a level priority of each of the number of clients accessing the data block at the one or more nodes, and generates the node assignment of the node for the data block among the plurality of nodes further based on the plurality of load factors associated with the plurality of nodes and the level priority of each of the number of clients accessing the data block at the one or more nodes.

[0043] Clause 18. A computer program product as described in any one of Clauses 15 to 17, wherein for a data block among the plurality of data blocks, the existence time data associated with the existence time of the data block includes the number of clients accessing the data block at the one or more nodes during the previous time period and the total number of times the data block was accessed at the one or more nodes during the previous time period, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for the data block among the plurality of data blocks: for each of the number of clients accessing the data block at the one or more nodes during the previous time period, determine the network latency for the client to access the data block at the one or more nodes based on the network latency data associated with the network latency for the client to access the data block at the one or more nodes and the read latency of the one or more nodes.

[0044] Clause 19. A computer program product as described in any one of Clauses 15 to 18, wherein generating a node assignment for a node among the plurality of data blocks comprises: providing as input to a machine learning model: (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients accessing the data block at one or more nodes in the previous time period, (iii) the total number of times the data block was accessed at one or more nodes in the previous time period, and (iv) the network latency for each of the number of clients accessing the data block at one or more nodes; and receiving the node assignment for the data block among the plurality of nodes as output from the machine learning model.

[0045] Clause 20. A computer program product as described in any one of Clauses 15 to 19, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: perform the dynamic data allocation continuously or periodically for each of the plurality of data blocks.

[0046] These and other features and characteristics of this disclosure, as well as the operational methods and manufacturing economies of combination of related structural elements and parts, will become more apparent when considered in conjunction with the accompanying drawings, all of which form part of this specification, wherein similar reference numerals denote corresponding parts in the figures. However, it should be clearly understood that the drawings are for illustrative and descriptive purposes only and are not intended to be a definition of limitation on the disclosed subject matter. Attached Figure Description

[0047] Additional advantages and details are explained in more detail below with reference to the non-limiting exemplary embodiments shown in the illustrative accompanying drawings, in which:

[0048] Figure 1 It is a schematic diagram of a system for dynamically distributing data to nodes in a distributed system, based on some non-limiting implementation schemes or aspects;

[0049] Figure 2 It is based on some non-restrictive implementation schemes or aspects Figure 1 A schematic diagram of example components of one or more devices;

[0050] Figure 3 This is a flowchart illustrating a method for dynamically distributing data to nodes in a distributed system, based on some non-limiting embodiments or aspects; and

[0051] Figure 4 This is a schematic diagram illustrating an example movement of data blocks from the current node to the node assigned by the node, according to some non-limiting implementation or aspect. Detailed Implementation

[0052] For the purposes of the following description, the terms “end,” “upper,” “lower,” “right,” “left,” “vertical,” “horizontal,” “top,” “bottom,” “lateral,” “longitudinal,” and their derivatives should be associated with the orientation of the embodiments in the accompanying drawings. However, it should be understood that various alternative variations and sequences of steps may be employed in the embodiments, except where explicitly specified as the opposite. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary embodiments or aspects of the disclosed subject matter. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered as limitations.

[0053] It should be understood that, except as expressly specified otherwise, this disclosure may take various alternative variations and sequences of steps. It should also be understood that the specific devices and processes shown in the accompanying drawings and described in the following specification are merely exemplary and non-limiting embodiments or aspects. Therefore, specific dimensions and other physical characteristics relating to the embodiments or aspects disclosed herein should not be considered as limitations.

[0054] This document describes some non-limiting implementations or aspects in conjunction with thresholds. As used herein, satisfying a threshold can refer to a value that is greater than, more than, higher than, greater than or equal to, less than, less than, lower than, less than or equal to, or equal to a threshold.

[0055] The terms "aspect," "component," "element," "structure," "action," "step," "function," and "instruction" used herein should not be construed as critical or essential unless explicitly stated otherwise. Furthermore, as used herein, the article "a" is intended to include one or more items and is interchangeable with "one or more" and "at least one." Additionally, as used herein, the term "group" is intended to include one or more items (e.g., related items, unrelated items, combinations of related and unrelated items, etc.) and is interchangeable with "one or more" or "at least one." Where only one item is desired, the term "a" or similar language is used. Moreover, as used herein, the term "having" and similar terms are intended to be open-ended terms. Additionally, unless explicitly stated otherwise, the phrase "based on" is intended to mean "at least partially based on." Furthermore, a reference to an action "based on" a condition may indicate that the action is "in response to" that condition. For example, in some non-limiting embodiments or aspects, the phrases "based on" and "in response to" may refer to conditions that automatically trigger an action (e.g., specific operations of electronic devices such as computing devices, processors, etc.).

[0056] As used herein, the term "communication" can refer to the receiving, accepting, sending, transmitting, or providing of data (e.g., information, signals, messages, instructions, commands, etc.). For one unit (e.g., a device, system, component of a device or system, or a combination thereof) to communicate with another unit means that the first unit is able to receive information from and / or send information to the other unit, directly or indirectly. This can refer to a direct or indirect connection that is inherently wired and / or wireless (e.g., a direct communication connection, an indirect communication connection, etc.). Furthermore, two units can communicate with each other even if the transmitted information can be modified, processed, relayed, and / or routed between them. For example, the first unit can communicate with the second unit even if it passively receives information and does not actively send information to the second unit. As another example, the first unit can communicate with the second unit if at least one intermediate unit processes information received from the first unit and transmits the processed information to the second unit. In some non-limiting embodiments or aspects, a message can refer to a network packet (e.g., a data packet, etc.) that includes data. It should be understood that many other arrangements are possible.

[0057] As used herein, the term "computing device" can refer to one or more electronic devices configured to process data. In some examples, a computing device may include the necessary components for receiving, processing, and outputting data, such as a processor, display, memory, input devices, network interfaces, etc. A computing device can be a mobile device. As examples, mobile devices may include cellular phones (e.g., smartphones or standard cellular phones), portable computers, wearable devices (e.g., watches, glasses, lenses, clothing, etc.), personal digital assistants (PDAs), and / or other similar devices. A computing device can also be a desktop computer or other form of non-mobile computer.

[0058] As used herein, the term "server" may refer to or include one or more computing devices operated by or facilitating communication and processing by multiple parties in a network environment such as the Internet, but it should be understood that communication may be facilitated through one or more public or private network environments, and various other arrangements may be possible. Furthermore, multiple computing devices (e.g., servers, point-of-sale (POS) devices, mobile devices, etc.) communicating directly or indirectly in a network environment may constitute a "system." As used herein, references to "server" or "processor" may refer to a previously stated server and / or processor, different servers and / or processors, and / or combinations of servers and / or processors that perform a prior step or function. For example, as used in the specification and claims, a first server and / or first processor stated to perform a first step or function may refer to the same or different servers and / or processors stated to perform a second step or function.

[0059] As used herein, the term "system" may refer to one or more computing devices or a combination of computing devices (e.g., processor, server, client device, software application, components of such computing devices, etc.). As used herein, references to "device," "server," "processor," etc., may refer to a previously stated device, server, or processor, different servers or processors, and / or combinations of servers and / or processors, that is stated to perform a prior step or function. For example, as used in the specification and claims, a first server or first processor stated to perform a first step or a first function may refer to the same or different server or the same or different processor stated to perform a second step or a second function.

[0060] Numerous existing efforts exist to address the data allocation / storage problem in distributed systems. These existing methods attempt to solve the problem when data is initially stored. However, in many cases, the needs of clients using the data change over time, and the initial allocation / storage decisions gradually become suboptimal. Furthermore, as systems in a distributed cluster age, their performance or health begins to deteriorate, and due to the nature of distributed systems, this performance degradation may not be isolated or contained, impacting the efficiency of the full or total workload running on the cluster.

[0061] Non-limiting embodiments or aspects of this disclosure may provide methods, systems, and / or computer program products for performing the following operations: storing multiple data blocks at multiple nodes in a distributed system; receiving node health status data associated with multiple read latency associated with the multiple nodes from the multiple nodes; and performing dynamic data allocation for data blocks among the multiple data blocks, the dynamic data allocation comprising: receiving existence time data associated with the existence time of the data block in the distributed system from one or more nodes of the multiple nodes on which the data block was stored in a previous time period; and receiving data from each of a number of clients who accessed the data block at one or more nodes in a previous time period. Receive network latency data associated with the network latency used by the client to access the data block at one or more nodes; generate node assignments for the data block among a plurality of nodes based on (i) node health status data, (ii) time-of-existence data and (iii) network latency data for each of the number of clients accessing the data block at one or more nodes; determine whether the node assigned to the data block is different from the current node on which the data block is currently stored; and in response to determining that the node assigned to the data block is different from the current node on which the data block is currently stored, move the data block from the current node to the node assigned to the node.

[0062] In this manner, non-limiting embodiments or aspects of this disclosure can enable dynamic data allocation / movement of data blocks among nodes in a distributed cluster or system that automatically evolves based on the needs of clients accessing data blocks, the overall efficiency of individual nodes in the cluster, and / or the impact of slower nodes on the overall cluster operation, thereby improving the overall utilization of the cluster. For example, the data allocator can connect to each node in the distributed system or cluster and collect information and / or data, such as the health status of the node, the lifetime of the data stored on the node, and / or the network latency associated with a client accessing the data on the node. As an example, after requesting data from a node, each client can establish a connection with the data allocator to transmit details of the accessed data and the time consumed to complete the request, in order to estimate the network latency involved in the request. Based on the health status of the node, the lifetime of the data stored on the node, and / or the network latency associated with a client accessing the data on the node, the data allocator can generate node assignments (e.g., node identifiers, etc.) for each data block in the data blocks stored in the distributed system or cluster.

[0063] Now for reference Figure 1 , Figure 1 This is a schematic diagram of a system 100 for dynamically distributing data to nodes in a distributed system, based on some non-limiting embodiments or aspects. For example... Figure 1 As shown, system 100 may include a data distributor system 102, multiple nodes—node 1, node 2, node 3, ... node N—and / or a client system 104. The data distributor system 102, the multiple nodes—node 1, node 2, node 3, ... node N—and / or the client system 104 may be interconnected via wired connections, wireless connections, or a combination of wired and wireless connections (e.g., establishing connections for communication).

[0064] The data distributor system 102 may include one or more devices capable of receiving information and / or data from multiple nodes—node 1, node 2, node 3, ... node N and / or client system 104—and / or transmitting information and / or data to multiple nodes—node 1, node 2, node 3, ... node N and / or client system 104—e.g., via a communication network. For example, the data distributor system 102 may include computing devices, such as servers, server clusters, and / or other similar devices.

[0065] Multiple nodes—node 1, node 2, node 3, ... node N—may include one or more devices capable of receiving information and / or data from data distributor system 102 and / or client system 104 and / or transmitting information and / or data to data distributor system 102 and / or client system 104, for example, via a communication network. For instance, multiple nodes—node 1, node 2, node 3, ... node N—may be implemented in a distributed system, wherein individual nodes among the multiple nodes—node 1, node 2, node 3, ... node N—may be implemented within a single device and / or system or distributed across multiple devices and / or systems in the distributed system. For instance, the nodes among the multiple nodes—node 1, node 2, node 3, ... node N—may include computing devices and / or be implemented by computing devices, such as servers, server clusters, and / or other similar devices. As an example, one or more of the multiple nodes—node 1, node 2, node 3, ... node N (e.g., one or more of the multiple nodes—node 1, node 2, node 3, ... node N, etc.—or one or more computing devices included in the multiple nodes—node 1, node 2, node 3, ... node N, etc.)—may be located in a different physical location than one or more other nodes—node 1, node 2, node 3, ... node N (e.g., one or more other nodes—node 1, node 2, node 3, ... node N, etc.—or one or more other computing devices included in the multiple nodes—node 1, node 2, node 3, ... node N, etc.)—.

[0066] In some non-limiting embodiments or aspects, the multiple nodes—node 1, node 2, node 3, ... node N—include multiple heterogeneous nodes. For example, different nodes among the multiple nodes—node 1, node 2, node 3, ... node N—may include different types of hardware, firmware, or combinations of hardware and software (e.g., different types and / or speeds of random access memory (RAM), different types and / or speeds of processors, etc.).

[0067] Client system 104 may include one or more devices capable of receiving information and / or data from data distributor system 102 and / or multiple nodes—node 1, node 2, node 3, ... node N—and / / (e.g., via a communication network) transmitting information and / or data to data distributor system 102 and / or multiple nodes—node 1, node 2, node 3, ... node N. For example, client system 104 may include computing devices such as servers, server clusters, and / or other similar devices.

[0068] Multiple nodes—node 1, node 2, node 3, ... node N—can be configured to store multiple data blocks. For example, data distributor system 102 can be configured to store multiple data blocks at multiple nodes—node 1, node 2, node 3, ... node N. As an example, such as Figure 1 As shown, example data blocks D11, D12, D13, ... D1N can be stored (e.g., initial storage, current storage, etc.) at node 1, example data blocks D21, D22, D23, ... D2N can be stored (e.g., initial storage, current storage, etc.) at node 2, example data blocks D31, D32, D33, ... D3N can be stored (e.g., initial storage, current storage, etc.) at node 3, and / or example data blocks DN1, DN2, DN3, ... DNN can be stored (e.g., initial storage, current storage, etc.) at node N. Client system 104 can be configured to access multiple data blocks at multiple nodes—node 1, node 2, node 3, ... node N. Multiple nodes—node 1, node 2, node 3, ... node N—can be configured to enable or allow multiple different clients or client systems 104 to access requested data blocks from among the multiple data blocks. In this example, one or more clients or client systems 104 (e.g., one or more clients or one or more computing devices of client system 104, etc.) may be located in a different physical location than one or more other clients or client systems 104 (e.g., one or more other clients or one or more other computing devices of client system 104, etc.). In some non-limiting embodiments or aspects, the client's data access pattern may be assumed to be uniform at a certain frequency.

[0069] A data block may include a sequence of bits or bytes, which may contain an integer number of records with a maximum length (e.g., block size). For example, a data block may include a sequence of data in bit or byte form that can be transmitted as a whole. Multiple data blocks may include various types of data. In some non-limiting embodiments or aspects, multiple data blocks may include transaction data associated with multiple transactions processed in an electronic payment network. For example, transaction data may include parameters associated with the transaction, such as account identifiers (e.g., PAN), transaction amount, transaction date and time, type of product and / or service associated with the transaction, currency exchange rate, currency type, merchant type, merchant name, merchant location, transaction approval (and / or rejection) rate, etc.

[0070] As mentioned in this article Figure 3In more detail, the data allocator system 102 can be configured to generate node assignments for data blocks among multiple nodes—node 1, node 2, node 3, ... node N—based on information and / or data received from multiple nodes—node 1, node 2, node 3, ... node N and / or client system 104, for each data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks). The data allocator system 102 can be configured to: determine whether the node assigned to the node is different from the current node on which the data block is currently stored; and / or, in response to determining that the node assigned to the node is different from the current node on which the data block is currently stored, move the data block from the current node to the node assigned to the node. For example, the data allocator system 102 can be configured to move a data block stored on a node (e.g., node 3, etc.) to another node (e.g., node 1, etc.). As an example, the data allocator system 102 can be configured to remove or delete a data block stored on a node (e.g., node 3, etc.) and store the data block on another node (e.g., node 1, etc.).

[0071] A communication network may include one or more wired and / or wireless networks. For example, a communication network may include cellular networks (e.g., Long Term Evolution (LTE) networks, third-generation (3G) networks, fourth-generation (4G) networks, fifth-generation (5G) networks, Code Division Multiple Access (CDMA) networks, etc.), Public Land Mobile Networks (PLMNs), Local Area Networks (LANs), Wide Area Networks (WANs), Metropolitan Area Networks (MANs), telephone networks (e.g., Public Switched Telephone Networks (PSTN)), private networks, self-organizing networks, intranets, the Internet, fiber-optic networks, cloud computing networks, etc., and / or combinations of these or other types of networks.

[0072] Now for reference Figure 2 The diagram illustrates example components of device 200 according to a non-limiting embodiment. As an example, device 200 may correspond to a data distributor system 102, one or more nodes—node 1, node 2, node 3, ... node N—and / or a client system 104. In some non-limiting embodiments, such a system or device may include at least one device 200 and / or at least one component of device 200. The number and arrangement of components shown are provided as examples. In some non-limiting embodiments, device 200 may include additional components, fewer components, different components, or components arranged differently compared to those shown. Additionally or alternatively, a set of components (e.g., one or more components) of device 200 may perform one or more functions described as being performed by another set of components of device 200.

[0073] like Figure 2As shown, device 200 may include bus 202, processor 204, memory 206, storage unit 208, input unit 210, output unit 212, and communication interface 214. Bus 202 may include components that allow communication between components of device 200. In some non-limiting embodiments, processor 204 may be implemented in hardware, firmware, or a combination of hardware and software. For example, processor 204 may include processors (e.g., central processing unit (CPU), graphics processing unit (GPU), accelerated processing unit (APU), etc.), microprocessors, digital signal processors (DSPs), and / or any processing unit that can be programmed to perform functions (e.g., field-programmable gate array (FPGA), application-specific integrated circuit (ASIC), etc.). Memory 206 may include random access memory (RAM), read-only memory (ROM), and / or another type of dynamic or static storage device (e.g., flash memory, magnetic memory, optical memory, etc.) that stores information and / or instructions for use by processor 204.

[0074] Continue to refer to Figure 2 Storage component 208 may store information and / or software related to the operation and use of device 200. For example, storage component 208 may include a hard disk (e.g., magnetic disk, optical disk, magneto-optical disk, solid-state disk, etc.) and / or another type of computer-readable medium. Input component 210 may include components that allow device 200 to receive information, such as via user input (e.g., touch screen display, keyboard, keypad, mouse, buttons, switches, microphone, etc.). Alternatively, input component 210 may include sensors for sensing information (e.g., Global Positioning System (GPS) components, accelerometers, gyroscopes, actuators, etc.). Output component 212 may include components that provide output information from device 200 (e.g., display, speaker, one or more light-emitting diodes (LEDs), etc.). Communication interface 214 may include transceiver-like components (e.g., transceivers, separate receivers and transmitters, etc.) that enable device 200 to communicate with other devices, such as via wired connections, wireless connections, or a combination of wired and wireless connections. Communication interface 214 may allow device 200 to receive information from another device and / or provide information to another device. For example, the communication interface 214 may include an Ethernet interface, an optical interface, a coaxial interface, an infrared interface, a radio frequency (RF) interface, a universal serial bus (USB) interface, a Wi-Fi® interface, a cellular network interface, etc.

[0075] Device 200 can perform one or more processes described herein. Device 200 can perform these processes based on processor 204 executing software instructions stored on a computer-readable medium, such as memory 206 and / or storage unit 208. The computer-readable medium can include any non-transient memory device. Memory devices include memory space located within a single physical storage device or memory space extended across multiple physical storage devices. Software instructions can be read into memory 206 and / or storage unit 208 via communication interface 214 from another computer-readable medium or from another device. When executed, the software instructions stored in memory 206 and / or storage unit 208 can cause processor 204 to perform one or more processes described herein. Additionally or alternatively, hardwired circuitry can be used in place of or in conjunction with the software instructions to perform one or more processes described herein. Therefore, the embodiments described herein are not limited to any particular combination of hardware circuitry and software. The term “programmed or configured” as used herein refers to the arrangement of software, hardware circuitry, or any combination thereof on one or more devices.

[0076] Now for reference Figure 3 The diagram shows a flowchart of a method 300 for dynamically distributing data to nodes in a distributed system, according to some non-limiting embodiments or aspects. Figure 3 The steps shown are for illustrative purposes only. It will be understood that in some non-limiting embodiments or aspects, additional, fewer, different, and / or different orders of steps may be used. In some non-limiting embodiments or aspects, steps may be performed automatically in response to the execution and / or completion of previous steps.

[0077] like Figure 3 As shown, at step 302, method 300 includes storing multiple data blocks at multiple nodes. For example, data allocator system 102 may store multiple data blocks at multiple nodes in a distributed system—node 1, node 2, node 3, ... node N. As an example, multiple nodes—node 1, node 2, node 3, ... node N—may store multiple data blocks. In this example, multiple data blocks may be stored (e.g., initial storage, current storage, etc.) at multiple nodes—node 1, node 2, node 3, ... node N.

[0078] like Figure 3As shown, at step 304, method 300 includes receiving node health status data associated with multiple read latency associated with multiple nodes. For example, data distributor system 102 may receive node health status data associated with multiple read latency associated with multiple nodes—node 1, node 2, node 3, ... node N. As an example, data distributor system 102 may receive node health status data associated with multiple read latency associated with multiple nodes—node 1, node 2, node 3, ... node N from multiple nodes—node 1, node 2, node 3, ... node N.

[0079] Node health status data may include, and / or the data allocator system 102 may determine, based on the node health status data, read latency (e.g., the amount of time a node takes to serve a request for access to a data block at that node, etc.). For example, node health status data may include, and / or the data allocator system 102 may determine, based on the node health status data, average read latency (e.g., the average amount of time a node takes to serve multiple requests from one or more clients, etc., due to node processing). As an example, for each of a plurality of nodes—node 1, node 2, node 3, ... node N—node health status data associated with multiple read latencyes of the plurality of nodes—node 1, node 2, node 3, ... node N—may include read latency, which includes the average amount of time consumed by that node in serving one or more requests for access to one or more data blocks at that node.

[0080] In some non-limiting embodiments or aspects, node health data may also include multiple load factors associated with multiple nodes. For example, a node's load factor may include the ratio of the number of data blocks stored at the node to the number of addresses within the node (e.g., the number of storage blocks, the number of storage locations, etc.). As an example, if each client is co-located and accesses the same or similar type of data, the data allocator 102 may attempt to place more data on a few selected nodes while using the remaining nodes to store significantly less data or no data at all. In this example, multiple load factors may be used as a forced override to instruct the data allocator system 102 (e.g., to the data allocator system 102's data allocation algorithm or model, etc.) to consider the load factors of individual nodes, thereby avoiding the placement of data that would cause individual nodes to violate specific load thresholds.

[0081] The health status of nodes that store and / or may potentially store data blocks can play a role in determining cluster utilization. For example, due to the nature of distributed systems, active (e.g., efficient, etc.) but suboptimally performing nodes may potentially impede the performance of the overall system workload and / or become bottlenecks. As an example, as a node's hardware system ages, the maximum clock speed of the processor may decrease, the disk of the hardware system may have sector failures, and / or similar conditions. Therefore, it may be advantageous to assess the health status of nodes before data allocation / storage, as allocating new, highly accessed data to nodes that cannot perform at sufficient or maximum capacity may be undesirable, but such nodes may be suitable for older, less frequently accessed data. For example, a non-limiting embodiment or aspect of this disclosure can systematically reduce the dependence of a distributed system or cluster on unhealthy / older nodes by making more use of unhealthy / older nodes for archiving purposes and moving more frequently accessed data to healthy / faster nodes.

[0082] like Figure 3 As shown, at step 306, method 300 includes receiving existence time data associated with the existence time of a data block. For example, data allocator system 102 may receive existence time data associated with the existence time of a data block in a distributed system. As an example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), data allocator system 102 may receive existence time data associated with the existence time of the data block in the distributed system from one or more of a plurality of nodes—node 1, node 2, node 3, ... node N—on which the data block was stored in a previous time period.

[0083] In some non-limiting embodiments or aspects, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), the existence time data associated with the existence time of the data block includes the number of clients accessing the data block at one or more nodes in a previous time period, the total number of times the data block was accessed at one or more nodes in a previous time period (e.g., the number of times the data block was accessed globally, etc.), the amount of time since the data block was first stored or stored at multiple nodes in the distributed system—node 1, node 2, node 3, ... node N, or a combination thereof.

[0084] In some non-limiting embodiments or aspects, multiple data blocks can be divided into several generations based on their existence time. For example, nodes or groups of nodes among multiple nodes—node 1, node 2, node 3, ... node N—can be designated or given priority to store a specific generation of data blocks.

[0085] The lifetime of a data block can be correlated with the frequency to which it is expected to be used. For example, more recent data may be accessed more frequently than older data. In this way, the lifetime of a data block can be relevant to allocation / storage decisions, as it may be advantageous to place newer data on nodes with higher or maximum computing power and / or provide lower or minimum access latency to clients, and as data blocks age, it may be advantageous to move less recent data to other nodes (e.g., slower nodes) to make room for more new data on faster nodes. For example, as the lifetime of a distributed system and / or its nodes increases, it may be advantageous to purge newer and / or more frequently accessed data from older nodes and sweep older and / or less frequently accessed data into those older nodes.

[0086] like Figure 3 As shown, at step 308, method 300 includes receiving network latency data associated with one or more clients accessing a data block. For example, data distributor system 102 may receive network latency data associated with one or more clients accessing a data block (e.g., one or more client systems 104, etc.). As an example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), data distributor system 102 may receive network latency data associated with the network latency of that client for accessing the data block at one or more nodes from each of a number of clients who accessed the data block at one or more nodes in a previous time period. In this example, after a request to access a data block at one of the nodes—node 1, node 2, node 3, ... node N—is completed (e.g., after accessing the data block at the node), the client system 104 can automatically transmit network latency data to the data distributor system 102. This network latency data includes information and / or data associated with the accessed data block (e.g., an identifier associated with the data block) and / or the node on which the data block is stored (e.g., an identifier associated with the node) and / or the amount of time consumed to access the data block at the node (e.g., the amount of time consumed to complete the request).

[0087] Network latency data may include, and / or the data allocator system 102 may determine, based on the network latency data, the network latency including the latency (e.g., the amount of time) introduced or consumed when serving a client's request to access a data block at a node due to network communication. For example, network latency data may include, and / or the data allocator system 102 may determine, based on the network latency data, the average network latency including, the average latency (e.g., the average amount of time) introduced or consumed when serving one or more requests to access a data block at a node due to network communication. As an example, network latency may be equal to the total latency (e.g., the total amount of time consumed by the client to access a data block at a node, etc.) minus the node's read latency (e.g., the amount of time the node uses to serve the request, the amount of time consumed due to node processing, etc.). In this example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), for each of the number of clients that accessed the data block at one or more nodes in a previous time period, the data allocator system 102 can determine the network latency for that client to access the data block at one or more nodes based on network latency data associated with the network latency for that client to access the data block at one or more nodes and the read latency of one or more nodes.

[0088] In some non-limiting embodiments or aspects, for a data block among multiple data blocks, the network latency data also includes a level priority for each of the number of clients accessing that data block at one or more nodes. For example, one or more clients or client systems 104 (e.g., one or more clients or one or more computing devices of client system 104, etc.) may be associated with a different level priority than one or more other clients or client systems 104 (e.g., one or more other clients or one or more other computing devices of client system 104, etc.). As an example, network latency data associated with clients having a higher priority level (e.g., network latency associated with that network latency data, etc.) may be more heavily weighted than network latency data associated with clients having a lower priority level (e.g., network latency associated with that network latency data, etc.) to generate node assignments as described herein.

[0089] For requests that must travel long distances, network latency can be high. Therefore, placing data physically closer to clients more likely to access it can be advantageous. However, because different clients may access the same data from different locations, calculating allocation / storage to reduce or minimize latency can become unusual, but not intractable. For example, allocating new or frequently accessed data to a higher-performance node farther from the client's location may perform worse than allocating it to a lower-performance node closer to the client's location. As an example, allocating new or frequently accessed data to a higher-performance node slightly farther from the client's location than to a lower-performance node slightly closer to the client's location may perform better than allocating it to a lower-performance node closer to the client's location. Furthermore, in cases where clients accessing data frequently are relocated to different physical locations relative to multiple nodes—node 1, node 2, node 3, ... node N—it may be advantageous for the data allocator system 102 to generate updated node assignments using updated network latency data associated with the clients at different physical locations. These updated node assignments help maintain similar or identical data access times for clients now at different physical locations.

[0090] As we can observe in the above scenario, when the client moves from one location at time t1 to another location at time t2, the data distributor is aware of this and relocates the data to the node closer to the client's new location, namely node 3.

[0091] like Figure 3 As shown, at step 310, method 300 includes generating node assignments for data blocks. For example, data allocator system 102 may generate node assignments for data blocks among multiple nodes—node 1, node 2, node 3, ... node N. As an example, for a data block among multiple data blocks (e.g., for each data block among multiple data blocks), data allocator system 102 may generate node assignments for the data block among multiple nodes based on (i) node health status data, (ii) time-of-presence data, and / or (iii) network latency data for each client among a certain number of clients accessing the data block at one or more nodes.

[0092] In some non-limiting embodiments or aspects, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), the data allocator system 102 may generate a node assignment for a node in a plurality of nodes for that data block by providing the following as input to a machine learning model: (i) a plurality of read latencies associated with the plurality of nodes, (ii) the number of clients accessing the data block at one or more nodes in a previous time period, (iii) the total number of times the data block was accessed at one or more nodes in a previous time period, (iv) for each of the number of clients accessing the data block at one or more nodes, the network latency of the client used to access the data block at one or more nodes, (v) a plurality of load factors associated with the plurality of nodes, (vi) the level priority of each of the number of clients accessing the data block at one or more nodes, any combination thereof, etc.; and receiving the node assignment for the data block in a plurality of nodes as output from the machine learning model.

[0093] In some non-limiting embodiments or aspects, the machine learning model may include a reinforcement learning model with feedback. For example, the data allocator system 102 may use reinforcement learning techniques to generate node assignments among multiple nodes for data blocks (e.g., for each data block). As an example, the data allocator system 102 may use reinforcement learning algorithms to generate machine learning models (e.g., predictive models, feedforward neural networks, feedback neural networks, etc.). For example, the data allocator system 102 may generate machine learning models based on (i) node health status data, (ii) time-of-presence data, and / or (iii) network latency data. In this example, the machine learning model can be designed to receive the following as inputs: (i) multiple read latencies associated with multiple nodes—node 1, node 2, node 3, ... node N; (ii) the number of clients accessing the data block at one or more nodes in a previous time period; (iii) the total number of times the data block was accessed at one or more nodes in a previous time period; (iv) the network latency for each of the number of clients accessing the data block at one or more nodes; (v) multiple load factors associated with multiple nodes—node 1, node 2, node 3, ... node N; (vi) the level priority of each of the number of clients accessing the data block at one or more nodes, any combination thereof; and provide node assignments (e.g., one-hot encodings identifying the nodes assigned to the data block) as outputs for the nodes of the data block among the multiple nodes. For example, for a five-node cluster, the node assignment for the first node in the five-node cluster for a data block may include the following array [1, 0, 0, 0, 0], which indicates that the data block should be stored on the first node of the five nodes in the cluster or distributed system.

[0094] Data allocator system 102 can store machine learning models (e.g., store the model for use). Data allocator system 102 can store machine learning models in data structures (e.g., databases, memory, etc.). As an example, the data structure can be located inside or outside data allocator system 102 (e.g., remotely from the data allocator system). In such an example, data allocator system 102 can use the machine learning model to generate node assignments.

[0095] In this way, reinforcement learning models or algorithms based on feedback (such as cluster utilization, node health status data, time-to-existence data, network latency data, etc.) can dynamically change data allocation and / or continuously evolve data allocation, which can provide better utilization of the cluster (e.g., better utilization of multiple nodes - node 1, node 2, node 3, ... node N, etc.).

[0096] like Figure 3 As shown, at step 312, method 300 includes determining whether the node assignment for the data block is different from the current node on which the data block is currently stored. For example, data allocator system 102 may determine whether the node assignment for the data block is different from the current node on which the data block is currently stored. As an example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), data allocator system 102 may determine whether the node assigned to that data block is different from the current node on which the data block is currently stored.

[0097] like Figure 3 As shown, at step 314, method 300 includes moving a data block from the current node to the node-assigned node in response to determining that the node assigned by the node is different from the current node on which the data block is currently stored. For example, data allocator system 102 may move a data block from the current node to the node-assigned node in response to determining that the node assigned by the node is different from the current node on which the data block is currently stored. As an example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), data allocator system 102 may move the data block from the current node to the node-assigned node in response to determining that the node assigned to that data block is different from the current node on which the data block is currently stored. As an example and also referenced Figure 4 This diagram illustrates an example movement 400 of a data block from the current node to the node assigned to it. The data allocator system 102 can remove or delete a data block (e.g., D33) stored on the current node (e.g., node 3, etc.) and store the data block (e.g., D33, etc.) on the node assigned to it (e.g., node 1, etc.). In this example, after moving the data block from the current node to the node assigned to it, the process can return to steps 304, 306, 308, or 310 of method 300, allowing the data allocator system 102 to receive feedback from multiple nodes—node 1, node 2, node 3, ... node N—and / / or the client system 104 and / or dynamically update the data allocation based on the received feedback.

[0098] like Figure 3As shown, at step 316, method 300 includes maintaining the data block at the current node in response to determining that the node assigned by the node is the same node as the current node on which the data block is currently stored. For example, data allocator system 102 may maintain the data block at the current node in response to determining that the node assigned by the node is the same node as the current node on which the data block is currently stored. As an example, for a data block in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), data allocator system 102 may maintain the data block at the current node in response to determining that the node assigned to that data block is the same node as the current node on which the data block is currently stored. In this example, after maintaining the data block at the current node, processing may return to steps 304, 306, 308, or 310 of method 300, such that data allocator system 102 may receive feedback from a plurality of nodes—node 1, node 2, node 3, ... node N and / or client system 104—and / or dynamically update the data allocation based on the received feedback.

[0099] In some non-limiting embodiments or aspects, for data blocks in a plurality of data blocks (e.g., for each data block in a plurality of data blocks, etc.), the data allocator system 102 may continuously or periodically perform data allocation including one or more steps of steps 304 to 316 of method 300. For example, the data allocator system 102 may (i) continuously or periodically receive updated node health status data associated with multiple read delays associated with the plurality of nodes—node 1, node 2, node 3, ... node N—from a plurality of nodes—node 1, node 2, node 3, ... node N; (ii) continuously or periodically receive updated existence time data associated with the existence time of the data block in the distributed system from one or more of the plurality of nodes—node 1, node 2, node 3, ... node N on which the data block was stored in a previous time period; (iii) continuously or periodically receive updated network latency data associated with the network latency used by the client to access the data block at one or more nodes from each of the number of clients that accessed the data block at one or more nodes in a previous time period; or any combination thereof. As an example, in response to receiving updated node health status data, updated time-of-existence data, updated network latency data, or any combination thereof, the data allocator system 102 may perform data allocation continuously or periodically, which includes generating node assignments (e.g., node assignments for each data block, etc.) among multiple nodes based on the updated node health status data, updated time-of-existence data, updated network latency data, or any combination thereof.

[0100] In this example, after multiple data blocks are assigned to multiple nodes—node 1, node 2, node 3, ... node N—and stored on those nodes (e.g., after multiple data blocks are initially assigned to multiple nodes—node 1, node 2, node 3, ... node N—and stored on those nodes, or after multiple data blocks are assigned to multiple nodes—node 1, node 2, node 3, ... node N—and stored on those nodes in the current data allocation / storage configuration, etc.), feedback from the multiple nodes—node 1, node 2, node 3, ... node N—and / / or the client system 104 to the data allocator system 102 can be received continuously, periodically, and / or in response to client access to stored data, and the data allocator system 102 can generate updated or new node assignments for new or updated nodes for each data block (e.g., for each data block, etc.) based on this feedback.

[0101] Although the embodiments have been described in detail for illustrative purposes, it should be understood that such details are for that purpose only, and this disclosure is not limited to the disclosed embodiments or aspects, but rather is intended to cover modifications and equivalent arrangements within the spirit and scope of the appended claims. For example, it should be understood that this disclosure contemplates, as far as possible, that one or more features of any embodiment or aspect may be combined with one or more features of any other embodiment or aspect.

Claims

1. A method comprising: storing, with at least one processor, a plurality of data blocks at a plurality of nodes in a distributed system; receiving, with the at least one processor, from the plurality of nodes, node health data associated with a plurality of read latencies associated with the plurality of nodes; and for a data block of the plurality of data blocks, performing, with the at least one processor, dynamic data placement comprising: receiving, from one or more nodes of the plurality of nodes on which the data block was stored in a previous time period, presence time data associated with a presence time of the data block in the distributed system; receiving, from each of a certain number of clients that accessed the data block at the one or more nodes in the previous time period, network latency data associated with a network latency of the client to access the data block at the one or more nodes; generating, based on (i) the node health data, (ii) the presence time data, and (iii) the network latency data for each of the number of clients that accessed the data block at the one or more nodes, a node assignment of a node of the plurality of nodes for the data block; determining whether the node of the node assignment for the data block is different from a current node on which the data block is currently stored; and in response to determining that the node of the node assignment for the data block is different from the current node on which the data block is currently stored, moving the data block from the current node to the node of the node assignment.

2. The method of claim 1, wherein for each node of the plurality of nodes, the node health data associated with the plurality of read latencies associated with the plurality of nodes comprises a read latency comprising an average amount of time consumed by the node to service one or more requests for access to one or more data blocks at the node.

3. The method of claim 2, wherein the node health data further comprises a plurality of load factors associated with the plurality of nodes, and wherein for the data block of the plurality of data blocks: the network latency data further comprises a tier priority of each of the number of clients that accessed the data block at the one or more nodes, and generating the node assignment of the node of the plurality of nodes for the data block is further based on the plurality of load factors associated with the plurality of nodes and the tier priority of each of the number of clients that accessed the data block at the one or more nodes.

4. The method of claim 2, wherein for the data block of the plurality of data blocks, the presence time data associated with the presence time of the data block comprises a number of clients that accessed the data block at the one or more nodes in the previous time period and a total number of times that the data block was accessed at the one or more nodes in the previous time period.

5. The method of claim 4, further comprising: For the data block of the plurality of data blocks, with the at least one processor: For each client of the number of clients that accessed the data block at the one or more nodes in the prior time period, determine the network latency of the client for accessing the data block at the one or more nodes based on the network latency data associated with the network latency of the client for accessing the data block at the one or more nodes and the read latency of the one or more nodes.

6. The method of claim 5, wherein generating, for the data block of the plurality of data blocks, the node assignment of the node of the plurality of nodes for the data block comprises: providing as input to a machine learning model: (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients that accessed the data block at the one or more nodes in the prior time period, (iii) the total number of accesses of the data block at the one or more nodes in the prior time period, and (iv) for each client of the number of clients that accessed the data block at the one or more nodes, the network latency of the client for accessing the data block at the one or more nodes; and receiving the node assignment of the node of the plurality of nodes for the data block as output from the machine learning model.

7. The method of claim 1, further comprising: continuously or periodically, with the at least one processor, performing the dynamic data placement for each data block of the plurality of data blocks.

8. A system comprising: at least one processor coupled to memory and configured to: store a plurality of data blocks at a plurality of nodes in a distributed system; receive, from the plurality of nodes, node health data associated with a plurality of read latencies associated with the plurality of nodes; and perform, for a data block of the plurality of data blocks, dynamic data placement, comprising: receive, from one or more nodes of the plurality of nodes on which the data block was stored in a prior time period, presence time data associated with a presence time of the data block in the distributed system; receive, from each client of a number of clients that accessed the data block at the one or more nodes in the prior time period, network latency data associated with a network latency of the client for accessing the data block at the one or more nodes; generate, based on (i) the node health data, (ii) the presence time data, and (iii) the network latency data for each client of the number of clients that accessed the data block at the one or more nodes, a node assignment of a node of the plurality of nodes for the data block; determine whether the node assignment for the data block is different from a current node on which the data block is currently stored; and if the node assignment for the data block is different from the current node on which the data block is currently stored, perform a data migration of the data block from the current node to the node of the plurality of nodes assigned to the data block. in response to determining that the node assignment for the data block is different from the current node on which the data block is currently stored, moving the data block from the current node to the node assigned the data block.

9. The system of claim 8, wherein, for each of the plurality of nodes, the node health data associated with the plurality of read latencies associated with the plurality of nodes comprises a read latency comprising an average amount of time consumed by the node to service one or more requests for access to one or more data blocks at the node.

10. The system of claim 9, wherein the node health data further comprises a plurality of load factors associated with the plurality of nodes, and wherein, for the data block of the plurality of data blocks: the network latency data further comprises a tier priority of each of the number of clients accessing the data block at the one or more nodes, and generating the node assignment of the plurality of nodes for the data block is further based on the plurality of load factors associated with the plurality of nodes and the tier priority of each of the number of clients accessing the data block at the one or more nodes.

11. The system of claim 9, wherein, for the data block of the plurality of data blocks, the time-to-live data associated with the data block comprises a number of clients accessing the data block at the one or more nodes in the previous time period and a total number of accesses to the data block at the one or more nodes in the previous time period.

12. The system of claim 11, wherein the at least one processor is further configured to: for the data block of the plurality of data blocks: for each of the number of clients accessing the data block at the one or more nodes in the previous time period, determine the network latency for the client to access the data block at the one or more nodes based on the network latency data associated with the network latency for the client to access the data block at the one or more nodes and the read latency of the one or more nodes.

13. The system of claim 12, wherein, for the data block of the plurality of data blocks, generating the node assignment of the plurality of nodes for the data block comprises: providing, as input to a machine learning model, (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients that accessed the data block at the one or more nodes in the prior time period, (iii) the total number of accesses to the data block at the one or more nodes in the prior time period, and (iv) for each of the number of clients that accessed the data block at the one or more nodes, the network latency used by the client to access the data block at the one or more nodes; and receiving, as output from the machine learning model, the node assignment of the node of the plurality of nodes for the data block.

14. The system of claim 8, wherein the at least one processor is further configured to: continuously or periodically perform the dynamic data allocation for each data block of the plurality of data blocks.

15. A computer program product comprising at least one non-transitory computer- readable medium including program instructions, the program instructions, when executed by at least one processor of a node of a plurality of nodes, cause the at least one processor to: store a plurality of data blocks at a plurality of nodes in a distributed system; receive, from the plurality of nodes, node health data associated with a plurality of read latencies associated with the plurality of nodes; and perform, for a data block of the plurality of data blocks, a dynamic data allocation, including: receive, from one or more nodes of the plurality of nodes on which the data block was stored in a prior time period, presence time data associated with a presence time of the data block in the distributed system; receive, from each of a number of clients that accessed the data block at the one or more nodes in the prior time period, network latency data associated with a network latency used by the client to access the data block at the one or more nodes; generate, based on (i) the node health data, (ii) the presence time data, and (iii) the network latency data for each of the number of clients that accessed the data block at the one or more nodes, a node assignment of a node of the plurality of nodes for the data block; determine whether the node assignment of the node for the data block is different from a current node on which the data block is currently stored; and in response to determining that the node assignment of the node for the data block is different from the current node on which the data block is currently stored, move the data block from the current node to the node assigned by the node assignment.

16. The computer program product of claim 15, wherein, for each node of the plurality of nodes, the node health data associated with the plurality of read latencies associated with the plurality of nodes comprises a read latency comprising an average amount of time consumed by the node to service one or more requests for access to one or more data blocks at the node.

17. The computer program product of claim 16, wherein the node health data further comprises a plurality of load factors associated with the plurality of nodes, and wherein, for the data block of the plurality of data blocks: the network latency data further comprises a tier priority of each client of the number of clients accessing the data block at the one or more nodes, and generating the node assignment of the nodes of the plurality of nodes for the data block is further based on the plurality of load factors associated with the plurality of nodes and the tier priority of each client of the number of clients accessing the data block at the one or more nodes.

18. The computer program product of claim 16, wherein, for the data block of the plurality of data blocks, the existence time data associated with the existence time of the data block comprises a number of clients accessing the data block at the one or more nodes in the prior time period and a total number of accesses to the data block at the one or more nodes in the prior time period, and wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: for the data block of the plurality of data blocks: for each client of the number of clients accessing the data block at the one or more nodes in the prior time period, determine the network latency of the client for accessing the data block at the one or more nodes based on the network latency data associated with the network latency of the client for accessing the data block at the one or more nodes and the read latency of the one or more nodes.

19. The computer program product of claim 18, wherein, for the data block of the plurality of data blocks, generating the node assignment of the nodes of the plurality of nodes for the data block comprises: providing as input to a machine learning model: (i) the plurality of read latencies associated with the plurality of nodes, (ii) the number of clients accessing the data block at the one or more nodes in the prior time period, (iii) the total number of accesses to the data block at the one or more nodes in the prior time period, and (iv) for each client of the number of clients accessing the data block at the one or more nodes, the network latency of the client for accessing the data block at the one or more nodes; and receiving the node assignment of the node of the plurality of nodes for the data block as output from the machine learning model.

20. The computer program product of claim 15, wherein the program instructions, when executed by the at least one processor, further cause the at least one processor to: continuously or periodically performing the dynamic data allocation for each data block of the plurality of data blocks.

Images