A system for optimizing storage replication in distributed data analysis systems using historical data access patterns

Abstract

Historical analysis of query patterns is used to discover relationships between data sets. These relationships are used to make optimal decisions about where to place data in a globally distributed environment, where data locality is an important factor in providing good query performance. A mixed integer programming model is used to solve a constraint-based system that balances the need to keep data local to other data and the cost of replicating data across low-bandwidth networks.

Description

[0001] CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application is a continuation of U.S. Patent Application No. 16 / 107,452 filed on August 21, 2018, the disclosure of which is incorporated herein by reference. Background Art

[0003] Cloud computing systems sometimes include a distributed data analysis engine that runs in multiple data centers distributed around the world. Each data center contains one or more servers. Users of such cloud computing systems can create organizations and projects. Within a project, the distributed data analysis engine allows users to create data sets and tables. Internally, tables are divided into data replication units called storage sets. Each storage set corresponds to one or more files stored on a server. Although users typically query their own data sets, it is also possible for a user to share a data set with another user, or to make it publicly available to many users. Multiple data sets can be merged together at query time, potentially requiring the system to read data from a large number of different data sets that may belong to any arbitrary user.

[0004] When evaluating a query, the distributed data analysis engine executes a set of processes within a specific server. These processes read the above storage set files and execute most efficiently when the files to be read are stored on the same server where the analysis process is running. Reading data from remote servers is inherently more expensive and involves higher latency. The amount of bandwidth available for cross-server data transfer is also limited and is a scarce resource. Due to these limitations, in some systems, cross-server reads beyond a very small limit will be prohibited, and all data processed by the analysis process must be present in the local server.

[0005] To provide fault tolerance, redundancy, and high availability, some systems replicate all save sets to each server where an analytical process can be run. However, replicating data to each server in use is problematic because increasing the number of servers in use is expensive in terms of the amount of data transfer required. The amount of data transfer increases linearly with each additional server, so it does not scale. Another problem with replicating data to each server is that it limits the growth of the system's storage capabilities because the system will be limited by the available storage size of the smallest server in use. Summary of the invention

[0006] The present disclosure provides a method for optimizing replicated data storage. The method includes: using one or more processors to identify data sets commonly owned by an organization; automatically storing the commonly owned data sets on the same computing device of a distributed computing system; using the one or more processors to analyze a query log of the distributed computing system; using the one or more processors to identify items with linked data sets in the query log, and using the one or more processors to store or replicate items with the most frequently linked data sets on the same computing device of the distributed computing system. In alternative methods, the data sets may have other relationships that result in common groupings.

[0007] According to some examples, the method may further include generating a graph using the one or more processors, wherein each identified item is represented by a node in the graph, and wherein the number of times two identified items are linked in the query log is represented by a weighted edge between the nodes representing the two items. Clusters of items connected by edges having weights greater than a predetermined threshold may be identified, wherein storing the items with the most linked data sets includes storing the identified clusters.

[0008] According to other examples, the method may include using a mixed integer programming model to identify clusters of projects based on linear relationships between the projects. The constraints of the mixed integer programming model may include storage and computing capabilities of each server, storage and computing requirements of each cluster, the cumulative storage and computing requirements of each cluster must not exceed the storage and computing capabilities of the servers assigned to the cluster, each cluster has multiple replicas, the replicas must be stored on separate servers, and data should be retained on servers that may already be located. An allocation graph may be generated based on the output of the mixed integer programming model, the allocation graph indicating where each cluster is stored. Data transfer operations required to implement the allocation graph may be determined to ensure that the available storage in the server is not exceeded when copying data from a source server to a target server.

[0009] Another aspect of the present disclosure provides a system for optimizing replicated data storage. The system includes one or more memories that store transaction logs between data sets in a distributed computing system over a period of time; one or more processors that communicate with the one or more memories. The one or more processors are configured to identify data sets that are commonly owned by an organization; automatically store the commonly owned data sets on the same computing device of the distributed computing system; analyze the logs of the distributed computing system; based on the analysis of the logs, identify items with linked data sets; and store items with the most frequently linked data sets on the same computing device of the distributed computing system.

[0010] Another aspect of the present disclosure provides a computer-readable medium storing instructions executable by a processor to perform a method for optimizing replicated data storage. The method includes: identifying a dataset commonly owned by an organization; automatically storing the commonly owned dataset on a same computing device of a distributed computing system; analyzing a query log of the distributed computing system; identifying items with linked datasets in the query log; and storing items with the most frequently linked datasets on a same computing device of the distributed computing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] Figure 1 is a block diagram of an example distributed computing system in accordance with aspects of the present disclosure.

[0012] Figure 2 is a block diagram of an example data distribution system in accordance with aspects of the present disclosure.

[0013] Figure 3 is a block diagram of an example allocation planner in accordance with aspects of the present disclosure.

[0014] Figure 4A-4B Example groupings of items in accordance with aspects of the present disclosure are shown.

[0015] Figure 5 is an example diagram illustrating dependencies of projects according to aspects of the present disclosure.

[0016] Figure 6 is a flow chart illustrating an example method according to aspects of the present disclosure. DETAILED DESCRIPTION

[0017] Overview

[0018] The present disclosure describes a system that uses historical analysis of query patterns to discover relationships between data sets. These relationships are used to make optimal decisions about where to place data in a globally distributed environment, where locality of data is an important factor in providing good query performance. A mixed integer programming model is used to solve a constraint-based system that balances the need to keep data local to other data and the cost of replicating data across a low-bandwidth wide area network (WAN).

[0019] According to one example implementation, the system includes an allocation planner and a replication controller. These entities can be modules within a computing device that perform specific processes, as described below. The allocation planner and replication controller work together to implement a replication strategy to replicate user data to a fixed number of servers out of a possible total number of available servers. By replicating only to a subset of servers rather than to every server, there is a fixed limit on the amount of replication required for each new storage set. The number of servers in the subset can be predetermined, for example, based on redundancy requirements. In addition, the total amount of storage required for the system is greatly reduced. By way of example only, in a system involving the use of 5 servers, previous replication techniques required 5 copies of each storage set. In contrast, the system described herein may choose to replicate to only 2 of the 5 servers. In this way, each server contains, on average, only 40% of the amount of data required by the previous system.

[0020] Replicating data to only a subset of servers results in the possibility of needing to read data stored on remote servers. While remote reads are possible, they can severely impact performance and are more throughput-constrained. To minimize these effects, the system minimizes the number of remote reads. In particular, data that might be merged by a user query is kept together, while any first user data can still be merged with a second user data.

[0021] To minimize the likelihood of remote requests, the system automatically places all storage sets owned by the same organization in the same server. Query log analysis is performed to identify historical data access patterns, which are used to predict future behavior. For example, the system will examine query logs to determine which data sets tend to appear together in queries most frequently and attempt to replicate those data sets to the same server. Users may have one or more organizations or projects, and they can organize their data within said organizations to keep their data available locally.

[0022] The allocation planner periodically queries the metadata catalog of the storage set to determine the amount of data stored by each project. For example, such an analysis can be performed every few hours, every few days, and so on. The allocation planner also queries the project / organization relationship and groups together all projects that are part of the same organization. It queries the job log to find instances where data from one project (or organization) is linked to data from another project (or organization). The query sums the amount of data transferred by queries performed in the past (such as in the recent past time period, such as the last few days) to assign weights to the relationships between projects.

[0023] Assuming that data sets should not be moved when possible, the allocation planner can use a greedy solver. In this example, a graph connected components algorithm can be used to cluster projects together based on these relationships. The projects represent nodes in a graph and the query history is used to construct weighted edges in the graph. The connected components algorithm finds clusters of projects connected by edges with weights greater than a certain threshold. These clusters represent related sets of projects and should be stored on the same server. In some examples, two different connected components passes are run. The first pass establishes relationships between projects based on sharing a common reservation, where a reservation is a set of pre-allocated computing resources that a customer can purchase and reserve for their own use. The second pass uses historical analysis of the query log to associate and merge projects together based on the query history.

[0024] In another example, the allocation planner can use a mixed integer programming model to solve the constraint-based problem of where to store each cluster. In this example, the relationships between related projects are expressed as linear constraints. The constraints of the mixed integer programming model may include: 1) Each server has storage and compute capabilities; 2) Each project cluster has a storage requirement and a compute requirement; 3) The sum of the storage and compute requirements of the project clusters assigned to a server cannot exceed the storage and compute capabilities of the server; 4) Each project cluster has multiple copies of the data, which must be stored on separate servers; 5) When any server is out of service or "exhausted", the remaining servers must assume the previous load on the exhausted server; 6) When feasible, data should be kept on servers that are already located to minimize data transfer costs. The model produces an allocation graph that maps each project cluster to a server. This is the output of the allocation planner.

[0025] The replication controller determines a replication plan based on the allocation map and the current state of the allocations. The replication plan identifies the data transfer operations required to achieve the new allocation map. This includes some number of data copy operations, as well as data deletion operations to reclaim unused space. The replication plan takes into account intermediate storage capacity so that even while data is being transferred, the available storage is not exceeded and it is possible to temporarily have more copies of data than is required in a static state. For example, while transferring project data from a source server to a target server, it can be temporarily copied to both the source and target servers. The replication controller ensures that both the source and target servers have sufficient storage capacity to accommodate the transfer.

[0026] Although the concepts described herein may be used to manage data replicated across multiple servers, it should be understood that such concepts may also be applied to other network architectures, such as those utilizing "cells." For example, a cell is a network of tightly connected computing devices that, for example, may run processes collaboratively, have shared storage, and have very low network latency and high throughput between computing devices in the cell. Thus, although the examples herein are primarily described with respect to assigning items to specific servers, it should be understood that these examples are similarly applicable to assigning items to cells or other computing devices or groups of computing devices.

[0027] Example System

[0028] Figure 1 An example system including a distributed computing environment is shown. Multiple data centers 160, 170, 180 can be communicatively coupled, for example, over a network 150. The data centers 160, 170, 180 can further communicate with one or more client devices (e.g., client 110) via the network 150. Thus, for example, client 110 can perform operations in the "cloud". In some examples, the data centers 160, 170, 180 can further communicate with a controller 190.

[0029] Data centers 160-180 may be located at a considerable distance from each other. For example, data centers may be located in various countries around the world. Each data center 160, 170, 180 may include one or more computing devices, such as processors, servers, shards, small elements, etc. For example, Figure 1 As shown, data center 160 includes computing devices 162, 164, data center 170 includes computing device 172, and data center 180 includes computing devices 181-186. Programs can be executed across these computing devices, for example, so that some operations are performed by one or more computing devices in a first data center, while other operations are performed by one or more computing devices in a second data center. In some examples, the computing devices in various data centers can have different capabilities. For example, different computing devices can have different processing speeds, workloads, etc. Although only a few of these computing devices are shown, it should be understood that each data center 160, 170, 180 can include any number of computing devices, and the number of computing devices in a first data center can be different from the number of computing devices in a second data center. Moreover, it should be understood that the number of computing devices in each data center 160-180 can change over time, for example, because hardware is removed, replaced, upgraded, or expanded.

[0030] In some examples, each data center 160-180 may also include multiple storage devices (not shown), such as hard drives, random access memory, disks, disk arrays, tape drives, or any other type of storage device. The data centers 162, 172, 182 may implement any of a variety of architectures and technologies, including but not limited to direct attached storage (DAS), network attached storage (NAS), storage area network (SAN), Fibre Channel (FC), Fibre Channel over Ethernet (FCoE), hybrid architecture networks, etc. In addition to storage devices, the data center may also include many other devices, such as cables, routers, etc. In addition, in some examples, the data centers 160-180 may be virtualized environments. In addition, although only a few data centers 160-180 are shown, multiple data centers may be coupled via the network 150 and / or additional networks.

[0031] In some examples, controller 190 can communicate with computing devices in data centers 160-180 and can facilitate the execution of programs. For example, controller 190 can track the capabilities, status, workload, or other information of each computing device and use such information to assign tasks. Similar to client 110 described above, controller 190 can include processor 198 and memory 192, memory 192 including data 194 and instructions 196. In other examples, such operations can be performed by one or more computing devices in one of data centers 160-180, and a separate controller can be omitted from the system.

[0032] Each client 110 can be a computer intended to be used by an individual or entity, for example. Client 110 can have all the internal components commonly found in a personal computer, such as a central processing unit (CPU), a CD-ROM, a hard disk drive, and a display device, such as a monitor with a screen, a projector, a touch screen, a small LCD screen, a television set or other device (such as an electronic device operable to display the information processed by the processor 120), a speaker, a modem and / or a network interface device, user input (such as a mouse, keyboard, touch screen or microphone), and all components for interconnecting these elements. In addition, a computer according to the system and method described herein can include a device capable of processing instructions and sending data to people and other computers and transmitting data from people and other computers, including a general-purpose computer, a PDA, a flat panel, a mobile phone, a smart watch, a network computer lacking local storage capacity, a set-top box of a television set, and other networking devices.

[0033] The client 110 may include a processor 120, a memory 130, and other components typically found in a general-purpose computer. The memory 130 may store information accessible to the processor 120, including instructions 132 that may be executed by the processor 120. The memory may also include data 134 that may be retrieved, manipulated, or stored by the processor 120. The memory 130 may be a non-transitory computer-readable medium of a type capable of storing information accessible to the processor 120, such as a hard drive, a solid-state drive, a tape drive, an optical memory, a memory card, a ROM, a RAM, a DVD, a CD-ROM, a memory with write capability, and a read-only memory. The processor 120 may be a well-known processor or other lesser-known types of processors. Alternatively, the processor 120 may be a dedicated controller, such as an ASIC.

[0034] The instructions 132 may be a set of instructions directly executed by the processor 120, such as machine code, or instructions indirectly executed, such as a script. In this regard, the terms "instructions," "steps," and "programs" may be used interchangeably herein. The instructions 132 may be stored in an object code format for direct processing by the processor 120, or in other types of computer languages, including scripts or collections of independent source code modules that are interpreted or pre-compiled as needed.

[0035] Data 134 may be retrieved, stored, or modified by processor 120 in accordance with instructions 132. For example, although the systems and methods are not limited to a particular data structure, data 134 may be stored in a computer register, in a relational database as a table with a plurality of different fields and records, or an XML document. Data 134 may also be formatted in a computer readable format, such as, but not limited to, binary values, ASCII, or Unicode. In addition, data 134 may include information sufficient to identify the relevant information, such as numbers, descriptive text, proprietary codes, pointers, references to data stored in other storage, including other network locations, or information used by functions for calculating the relevant data.

[0036] Application 136 can be used for any of a variety of operations. Application 136 can be downloaded, executable from instructions 132, or accessed remotely, for example. In some examples, the application can be executed remotely. For example, an application on a client device can be executed in the cloud.

[0037] although Figure 1The processor 120 and memory 130 are functionally shown as being within the same block, but the processor 120 and memory 130 may actually include multiple processors and memories that may or may not be stored within the same physical housing. For example, some instructions 132 and data 134 may be stored on a removable CD-ROM, while other instructions may be stored within a read-only computer chip. Some or all instructions and data may be stored in a location physically remote from the processor 120 but still accessible to the processor 120. Similarly, the processor 120 may actually include a collection of processors that may or may not operate in parallel.

[0038] Client 110, data center 160-180 and control 190 can communicate directly and indirectly, for example, through network 150. For example, using Internet sockets, client 110 can connect to a service operating on a remote server through the Internet protocol suite. The server can set a listening socket that can accept an initial connection for sending and receiving information. Network 150 and intermediate nodes can include various configurations and protocols, including the Internet, the World Wide Web, an intranet, a virtual private network, a wide area network, a local area network, a private network using a communication protocol proprietary to one or more companies, Ethernet, WiFi (e.g., 702.71, 702.71b, g, n or other such standards) and HTTP, and various combinations of the above. Such communication can be facilitated by devices that can send data to and from other computers, such as modems (e.g., dial-up, cable or fiber optic) and wireless interfaces.

[0039] Figure 2 An example architecture of a system for optimizing replicated storage is shown. The system may reside, for example, on Figure 1 On a computing device in one of data centers 160-180 Figure 1 , or some combination thereof. The system includes a load balancer 250, which can be responsible for allocating reservations to compute servers in a balanced manner. For example, the load balancer 250 can balance the distribution of tasks within a data center or between globally distributed systems. The load balancer 250 can, for example, compute an explicit project-to-server allocation mapping to achieve appropriate load balancing. It can handle both static query routing when all servers are available and dynamic query routing when servers are exhausted. In addition, the load balancer 250 can provide a reliable way to publish these allocations to all components in the data analysis engine that use this information.

[0040] The data set may be replicated two or more times, such as in a primary and a secondary replica. The load balancer 250 and routing system runs queries in the location with the primary replica. The secondary replica is a backup that becomes the active location for query routing if the primary server is exhausted. This has a greater impact on the system when replicas are bound to reservations with pre-allocated slots. Computing power may be reserved for the primary server, with the additional constraint that any single server may go down and the system can reroute queries to the secondary location without overloading any server.

[0041] The load balancer 250 may include an allocation planner 252, a replication controller 254, and a reservation exchange manager 256. The allocation planner 252 may run as a subcomponent in the load balancer 250. For example, the allocation planner 252 may be a module, such as a C++ or other type of module. In some cases, the allocation planner may replace an existing routing calculation component. Although in some examples, a single allocation planner 252 may generate allocations for each region, in other examples, additional or fewer allocation planners may be implemented. For example, the allocation planner may generate allocations for a globally distributed system, multiple allocation planners may be implemented in a given region, and the like.

[0042] The allocation planner 252 is responsible for calculating and issuing allocations. When calculating allocations, the allocation planner considers a variety of factors, such as the hierarchy of organizations and projects to be allocated (216), on-demand computing consumption, storage (e.g., by organization), server storage and computing power, etc. In some examples, the allocation planner may also consider computing reservations (214), such as resources guaranteed to specific customers. To obtain the latest reservation information, for example, the allocation planner 252 can subscribe to periodic updates, receive push notifications when new reservations are made, etc. The information considered by the allocation planner 252 when generating a new allocation plan can include historical query information, such as metadata snapshots 212, data sets, etc.

[0043] Allocation planner 252 may periodically obtain data from sources and predict future events, which are taken into account when determining allocations. For example, allocation planner 252 may predict rejections, customers about to send large amounts of data, potential reservation changes, etc. Such events may affect server size, project storage, or other constraints.

[0044] The allocation planner 252 may use a solver to calculate feasible allocations. The solver may be, for example, a greedy algorithm, a mixed integer problem solver, or any other type of solver. Some example solvers will be discussed in further detail below. It may further provide information about the calculated allocations to the replication controller 254. The replication controller 254 may be implemented as a library in the load balancer 250.

[0045] The allocation planner 252 can run continuously on a schedule, on demand, or in response to environmental changes or configuration changes. It can operate in a dry run mode, in which it generates allocations but does not apply them, or in an autonomous mode, in which it automatically sends newly generated allocations to the replication controller 254 for immediate application.

[0046] Once the new allocations are calculated, they may differ from the previous allocations. For example, a set of allocations, such as for a particular organization, may be moved from a first replica to a second replica. The replication controller 254 is responsible for moving data to conform to the new allocations and cleaning up data in the previous location. For example, the replication controller 254 may determine a replication plan for the data transfer operations necessary to implement the new allocations. The plan may include many data transfer and replication operations, as well as data deletion operations to reclaim unused space. The replication plan may take into account intermediate storage capabilities so that even during data transfers, the available storage is not exceeded and there are temporarily more copies of the data than are required in a static state.

[0047] According to some examples, the replication controller 254 can manage a state machine for each organization that needs to be moved to a new replica. In doing so, the replication controller 254 can start by adding the new replica to the replication configuration of the organization so that all new data begins to enter the new replica. The replication controller 254 further commands the move to ensure that there is enough space available to replicate the entire organization. It also calls the replication worker 264 to perform data replication. When the primary reservation of an organization needs to be moved between servers, the replication controller 254 submits a reservation move request to the reservation exchange manager (RSM) 256. For example, a reservation can be moved when the reservation is moved out of the target server. Once the data move is completed, the publishing mechanism of the load balancer 250 can push those changes to all servers involved in the routing decision.

[0048] The replication worker 264 may be a service that handles replication storage at different metadata granularities. In some examples, it may be created by splitting replication-specific functionality from the replication queue handler. It may be able to replicate a single storage set or all storage for a table or table range. It may also support deleting replicas.

[0049] The RSM 256 handles reservation moves and can perform reservation swaps when there is not enough room for one reservation to move between servers. For example, if there are two servers, each with 1000 available slots, and a reservation for 500 slots is to be moved from the first of the two servers to the second, it can be moved because resources are available. However, if reservations for 1500 and 2000 slots are to be moved at the same time, they are automatically swapped. For example, a reservation can publish its intent and when both reservations are ready, a swap is performed and both reservations are moved simultaneously.

[0050] According to some examples, each of allocation planner 252, replication controller 254, and reservation exchange manager 256 may export dashboards of status, metrics, and the like.

[0051] Figure 3 An example of the allocation planner 252 is shown in more detail. In this example, the allocation planner 252 includes multiple components, such as an input collector / analyzer 302, an input validator 304, one or more solvers 306, and an allocation validator 308. It should be understood that these components are merely examples and that additional, fewer, or different components may be included in the allocation planner 252.

[0052] Input collector / analyzer 302 reads project usage input from the metadata snapshot to determine the current size of the submitted data for each project. According to some examples, the data can be merged with project data from other sources to obtain organizational information for each project. Input collector / analyzer 302 also obtains reservation information. For example, the reservation information can be read from a storage file designated for storing static reservation information or from another source.

[0053] The input analyzer / collector 302 may also determine groupings of projects and organizations. According to one example, a hash strategy may be used to distribute small projects or organizations (e.g., small projects or organizations with 512GB or less of data) across servers. For example, small projects or organizations may be grouped into hash groups that are provided to the solver 306 for placement. The input analyzer / collector 302 may also determine hard groupings of data and create allocation groups that may be provided to the solver 306 as atomic units that cannot be split by the solver. For example, an allocation group may include any independent project, all projects in an organization, a hash bucket of a small organization or project, all projects in a reservation, and the like. All projects in an allocation group may be placed on the same server.

[0054] If there is a situation where a reservation includes projects from multiple organizations, the reservation's allocation group can be expanded to include all projects from any organization involved in the reservation. Figure 4AAs shown, reservation R includes projects P1 and P2 of organization A and projects P3 and P4 of organization B, but does not include all projects of organization B. Reservation R does not include project P5. Therefore, Figure 4B As shown, the allocation group of reservation R is expanded to include all projects in organization B. In particular, an allocation group G' is created that covers all groups in reservation R and any covered organizations. In this case, the allocation group will have the compute requirements of reservation R and the storage requirements of all included projects P1-P5. If project P5 is in a different reservation Q, then the G' group will cover all projects in both reservation R and Q. In this case, G' combines the compute requirements of reservations R and Q together.

[0055] The input analyzer / collector 302 may further determine the available capacity of each server. For example, it may receive an indication of such availability through an application programming interface (API) or any other direct or indirect communication with the server. Data for any input or output of the system may have a manual override. For example, if a server is intentionally underutilized or overutilized to allow for planned capacity changes, reservation changes are planned, or the size or computing requirements of a project are expected to grow in the near future, a manual override may be used. Data for manual overrides may be read from, for example, a configuration file and merged into inputs dynamically collected from other parts of the system.

[0056] The system can further compute dependencies. For example, the system can obtain a list of groups and create a weighted graph based on the query history. For example, the input collector / analyzer 302 can review a period of query history, such as the past few days, to determine which items are related. According to some examples, the system can generate a graph of related items.

[0057] Figure 5 An example graph of dependencies is shown. According to this example, a query running in project P reads 5GB from project Q and 10GB from project R. If the job needs to access information from projects Q, R, then each job executed by project P will produce one or more edges between node P and the related nodes Q, R. The edges for all jobs are added together to give a total weight (e.g., 5GB, 10GB).

[0058] Some projects may contain publicly visible or widely shared data that is used by a large number of users outside of the project that owns the data. These frequently accessed datasets may have a large number of edges and create "superclusters" of linked nodes. These projects can be excluded from the graph dependency analysis and automatically replicated to all nodes. For example, the system can detect a threshold count of edges for a given project and automatically replicate the project in full.

[0059] The input validator 304 can ensure that each input is correctly input to the system. For example, the input validator 304 can confirm that all required elements exist (e.g., each group has a valid, non-zero size), all IDs in the relationship graph exist in the input list, etc. According to some examples, the input validator 304 can also confirm the input related to the primary replica and the secondary replica. For example, the input validator 304 can confirm that the relationship between the primary replica and the secondary replica is valid (e.g., both have the same storage requirements, and only the primary replica represents the computational requirements), the primary replica and the secondary replica of the same group have an infinite weight, indicating that they should not be placed in the same server, and so on.

[0060] Solver 306 takes input and runs an allocation algorithm. Solver 306 may include a naive solver, such as a greedy solver. In some examples, solver 306 may further include an optimization solver, such as a mixed integer model. Solver 306 outputs an allocation graph that allocates each item to a server.

[0061] The allocation validator 308 operates on the allocations output from the solver 306. The validator 308 can be, for example, a C++ class. The validator 308 verifies that the allocation set is the correct solution for the given input. It verifies that no hard constraints are violated. For example, it can verify that no server is over-allocated in terms of compute or storage. The allocation validator 306 can also verify that all inputs are mapped to a single location. The validator 308 can also attempt to analyze the "quality" of the solution in terms of the amount of data movement required to implement the solution from the previous state and how the allocation is balanced across the available servers. For example, the validator 308 can assign a grade or score to each specific solution. The determination of the validity and score of each solution can be stored in a distributed system.

[0062] As described above, the solver 306 can execute one or more algorithms to generate the allocation. One example includes a greedy algorithm. For example, the input to the process may include a list of n servers C1...Cn with storage capabilities CS1...CSn and computing capabilities CC1...CCn, and a list of 2m groups G1...G2m with storage capabilities GS1...GS2m and computing capabilities GC1...GC2m. For each group G, there may be an allocation A(G)→C, which maps the group to the server. The input collector 302 produces m groups of items and then generates 2 inputs to the allocation solver for each group - a primary group and a secondary group, which is a backup copy of the primary data. Further inputs to the solver may be a list of m inverse edges from G1→G2, G3→G4…G2m-1→G2m (where weight is infinite), and a list of e edges E1…Ee from group Gi to Gj, where for each edge i= / j En, there is a non-zero positive weight W(En). The inverse edges represent the constraint that the primary and secondary replicas of a group should not be located on the same server. The edges represent data sharing relationships between items in a group based on empirical analysis of the task history.

[0063] The greedy solver can be used with a graph connected component (GCC) solver that takes a graph {N, E} of nodes N and edges E and returns a set of clusters L. Each cluster Ln is a set of groups from G. The storage requirement LSn is ΣGS of all groups in the cluster, and the computation requirement LCn is ΣGC of all groups in the cluster. The allocation function A(Ln) for a cluster returns the server with the maximum storage allocated to it among all groups in Ln. According to some examples, this can be implemented as a machine learning model.

[0064] A greedy solver may perform clustering on all groups with weights greater than some threshold. Initially, the threshold may be 0, so that all weights are considered in the first iteration. Clusters are sorted in descending order of compute requirements and then in descending order of storage requirements. The solver may attempt to bin pack the clusters first using existing allocations. If the existing allocations cannot accommodate, the cluster is placed in the server with the most compute power (if the cluster has compute requirements), otherwise it is placed in the server with the most available storage power. If there is no solution for the cluster, the solver increases the edge weight threshold, regenerates the clusters, and then reruns the bin packing algorithm. Example pseudocode for a greedy solver is as follows:

[0065]

[0066]

[0067] Greedy solvers favor the current allocation and will not move an existing cluster away from its currently allocated server as long as the server has the compute / storage capacity for that cluster. Data may be moved if clusters are merged together. For example, if Ci and Cj are linked to form Cij, this new cluster may have a different allocation. Assume that Ci is the larger of the two clusters and A(Ci) ≠ A(Cj). In this case, A(Cij) will be the same as the previous allocation A(Ci) and Cj will be moved.

[0068] Another example algorithm performed by solver 306 is a mixed integer problem. An example of a mixed integer problem includes the following constants and variables:

[0069] constant

[0070]

[0071] variable

[0072]

[0073] The model includes multiple constraints. For example, such constraints may include that each replica is assigned once, that certain replicas cannot be on the same server, that the server has a maximum capacity, bounds on movement variables or transfer cost variables, etc. Such constraints may be expressed using the above constants and variables. For example, for a constraint that can be expressed as each replica is assigned once, such that for all replicas i, sum_j x(i, j) = 1. For a constraint that can be expressed as certain replicas cannot be on the same server, such that for all replicas i and i', TRANSFER_COST(i, i') = -infinity, for all servers j, x(i, j) + x(i', j) <= 1. For servers with maximum capacity, for all server pairs, all primary replicas assigned to j and all secondary replicas assigned to j and whose primary replicas are assigned to j' cannot exceed the computing capacity of server j. This covers the case where server j' is down and the secondary replicas of the primary replica assigned to j' must take over the computing load. For all servers j and j' where j ≠ j', for all resources r, sum_i x(i,j)*USAGE(i,COMPUTE)*IS_PRIMARY(i)+sum_i'(x(i,j',i',j)*USAGE(i',COMPUTE)*IS_BACKUP(i',i,COMPUTE))<=CAPACITY(j,COMPUTE). In the bound transfer variable, for all i in the replica, m(i)>=1-x(i,PREVIOUS_SERVER(i)). In the bound transfer cost variable, for all i in the replica, j in the server, t(i,j)>=sum_i'sum_j'>jx(i',j')*TRANSFER_COST(i,i')-x(i,j)*(sum_i'TRANSFER_COST(i,i')). This constraint ignores replica i' for which TRANSFER_COST(i,i') equals -infinity and assumes that TRANSFER_COST(i,i') is always positive or NULL. The goal of the mixed integer solver is to minimize the move cost: minimize sum_im(i)*MOVE_COST(i)+sum_jt(j).

[0074] Example Method

[0075] Figure 6An example method 600 for generating a distribution of replicated data in a distributed system based on an analysis of historical queries is shown. The method can be performed, for example, at any device in a distributed system, such as a server, a controller, a load balancer, or other computing device. It should be understood that the following operations do not have to be performed in the exact order described below. Instead, the various steps can be processed in a different order or simultaneously. Unless otherwise specified, steps can also be added or omitted.

[0076] In block 610, data sets that are commonly owned by the same organization are identified. For example, an online retailer may be a cloud computing / storage customer. The online retailer may have multiple projects, for example, where each project is owned by a different team within the company. For example, the online retailer may have tables related to operations, such as sales orders or inventory in one project. The retailer may also have projects related to marketing or web analytics in a different project. The retailer may want to merge data together to find relationships between marketing efforts and order history.

[0077] In block 620, the jointly owned data sets are automatically stored on the same device in the distributed system. For example, the jointly owned data sets may be hard grouped, which will not be separated during subsequent group assignments. Referring to the online retailer example above, sales item information and marketing item information may be automatically stored together so that frequent association of items may be accomplished more efficiently. Joint ownership may be determined, for example, by comparing identifiers in the metadata, each identifier corresponding to a particular organization.

[0078] In box 630, for example, query logs for the distributed system are analyzed to identify relationships between data sets. For example, the query log may include metadata snapshots, compute reservations, information about the organization / project hierarchy, etc. The log may provide an indication of how a given project accesses information of another project in a different organization. For example, an online retailer may have one or more affiliates, such as an advertising agency, a marketing platform, affiliated websites, etc. The online retailer may need to access data from one or more affiliates, and vice versa, although one or more affiliates may store such data in different areas, such as different servers, different data centers, different regions, etc. Many other example implementations are also possible where data of multiple linked organizations is stored in the cloud or across distributed computing systems.

[0079] In block 640, items with linked data sets are identified by analyzing the query log. For example, the query log may reveal a history of transactions between the online retailer and one or more affiliated organizations over a period of time in the past. For example, the period of time may be a number of hours, days, weeks, etc. immediately prior to the time of analysis. Thus, the data sets of the online retailer and one or more affiliated organizations accessed in those transactions in the query log may be identified as linked.

[0080] In box 650, the items with the most frequently linked data sets are stored on the same computing device in the distributed system. For example, a solver can be used to determine the data sets that are most frequently accessed. Such as by generating a weighted graph or solving a mixed integer problem. This can also take into account the cost of each access, such as latency, priority, etc. In some examples, the data sets can be weighted, scored, ranked, or otherwise evaluated to determine those that are most frequently accessed and / or those data sets with the highest access costs. For example, an online retailer can access the affiliate advertising agency data 3 times in a historical time period, and the cost of each access is relatively low. The online retailer can also access the data of the affiliate mortgage analyst website 50 times in a historical time period, and the cost of each access is slightly higher. If the server only has enough capacity to store the data of two of the three organizations, they can store the data of the online retailer and the affiliate mortgage analyst together in the same device.

[0081] The advantage of the aforementioned technologies is that they provide improved data storage efficiency, which translates into improved computing and transaction efficiency. For example, by identifying data sets linked by transaction history and storing these data sets together, the delay in executing future transactions involving the two data sets can be reduced.

[0082] Unless otherwise stated, the aforementioned alternative examples are not mutually exclusive, but can be implemented in various combinations to achieve unique advantages. Since these and other variations and combinations of the features discussed above can be utilized without departing from the subject matter defined by the claims, the aforementioned description of the embodiments should be carried out in an illustrative manner rather than by the subject matter of the defined claims. In addition, the provision of the examples described herein and the phrases expressed as "such as", "including", etc. should not be interpreted as limiting the subject matter of the claims to specific examples. On the contrary, these examples are intended only to illustrate one of many possible embodiments. In addition, the same reference numerals in different drawings can identify the same or similar elements.

Claims

1. A method for optimizing replicated data storage, the method comprising: Use one or more processors to identify data sets that are jointly owned by the organizations; automatically storing the commonly owned data set on a same computing device in a distributed computing system; analyzing, using the one or more processors, a query log of the distributed computing system, wherein the query log indicates how a given project accessed a data set of another project in a different organization; identifying, using the one or more processors, projects in the query log that have linked data sets, including finding instances where a data set from one project is linked to a data set from a different project; The item having the most frequently linked data set is stored on the same computing device of the distributed computing system using the one or more processors.

2. The method according to claim 1, further comprising: generating a graph using the one or more processors, wherein each identified item is represented by a node in the graph, and Therein, the number of times two of the identified items are linked in the query log is represented by a weighted edge between nodes representing the two items. 3 . The method of claim 2 , further comprising identifying, using the one or more processors, clusters of items connected by edges having weights greater than a predetermined threshold.

4. The method according to claim 3, wherein: Storing the project with the most linked datasets includes storing the identified clusters.

5. The method according to claim 1, further comprising: Determine, using the one or more processors, data transfer operations required to implement storing the items having the most frequently linked data set on the same computing device, wherein determining the data transfer operations includes ensuring that available storage in the servers is not exceeded when copying data from a source server to a target server. 6 . The method of claim 1 , further comprising identifying clusters of items based on linear relationships between items using a mixed integer programming model.

7. The method according to claim 6, wherein: The mixed integer programming model is configured to solve the problem of where to store the clusters based on constraints, and the constraints of the mixed integer programming model include one or more of the following: storage and computing capabilities of each server, storage and computing requirements of each cluster, the cumulative storage and computing requirements of each cluster must not exceed the storage and computing capabilities of the servers allocated to the cluster, each cluster has multiple replicas that must be stored on separate servers, and data should be retained on servers that may already be located.

8. The method of claim 6 or claim 7, further comprising generating, using the one or more processors, an allocation map based on an output of the mixed integer programming model, the allocation map indicating where each cluster is stored.

9. The method of claim 8, further comprising determining, using the one or more processors, data transfer operations required to implement the allocation map.

10. The method according to claim 9, wherein: Determining the data transfer operation includes ensuring that available storage in the servers is not exceeded when copying data from the source server to the target server.

11. A system for optimizing replicated data storage, the system comprising: one or more memories storing a log of transactions between data sets in a distributed computing system over a period of time; one or more processors in communication with the one or more memories, the one or more processors configured to: Identify datasets that are commonly owned by the organizations; automatically storing the commonly owned data set on a same computing device in the distributed computing system; analyzing logs of the distributed computing system, wherein the logs indicate how a given project accessed a data set of another project in a different organization; Based on analysis of the log, identifying projects with linked data sets, including finding instances where a data set from one project is linked to a data set from a different project; The items having the most frequently linked data sets are stored on the same computing device of the distributed computing system.

12. The system according to claim 11, wherein: The one or more processors are further configured to: Generate graph wherein each identified item is represented by a node in the graph, and Therein, the number of times two of the identified items are linked in the log is represented by a weighted edge between nodes representing the two items.

13. The system according to claim 12, wherein: The one or more processors are further configured to identify clusters of items connected by edges having weights greater than a predetermined threshold.

14. The system according to claim 13, wherein: Storing the project with the most linked datasets includes storing the identified clusters.

15. The system according to claim 11, wherein: The one or more processors are further configured to identify clusters of items based on linear relationships between the items using a mixed integer programming model.

16. The system of claim 15, wherein: The mixed integer programming model is configured to solve the problem of where to store the clusters based on constraints, and the constraints of the mixed integer programming model include one or more of the following: storage and computing capabilities of each server, storage and computing requirements of each cluster, the cumulative storage and computing requirements of each cluster must not exceed the storage and computing capabilities of the servers assigned to the cluster, each cluster has multiple replicas that must be stored on separate servers, and data should be retained on servers that may already be located.

17. A system according to claim 15 or claim 16, wherein: The one or more processors are further configured to generate an allocation map based on an output of the mixed integer programming model, the allocation map indicating where to store each cluster.

18. The system of claim 17, wherein: The one or more processors are further configured to determine data transfer operations required to implement the allocation map.

19. The system of claim 18, wherein: Determining the data transfer operation includes ensuring that available storage in the servers is not exceeded when copying data from the source server to the target server.

20. A non-volatile computer-readable storage medium storing instructions, which when executed by a processor causes the processor to implement a method for optimizing replicated data storage, the method comprising: Identify datasets that are commonly owned by the organizations; automatically storing the commonly owned data set on a same computing device in a distributed computing system; analyzing a query log of the distributed computing system, wherein the query log indicates how a given project accessed a data set of another project in a different organization; identifying projects having linked data sets in the query log, including finding instances where a data set from one project is linked to a data set from a different project; The items having the most frequently linked data sets are stored on the same computing device of the distributed computing system.

Images