Master data placement in distributed storage systems

Abstract

Systems, methods, and computer-readable media for managing placement of data items on a distributed storage system. In some examples, a method can include determining a location of a primary copy of a data item on a distributed storage system, the location comprising a data store on the distributed storage system; determining an access pattern associated with the primary copy of the data item, the access pattern comprising originating locations of access requests for the primary copy of the data item received by the distributed storage system and a respective number of access requests received from each originating location; based on the access pattern, determining a different location on the distributed storage system for storing the primary copy of the data item, the different location comprising a different data store on the distributed storage system; and placing the primary copy of the data item at the different location.

Description

[0001] Cross-reference to related applications

[0002] This application claims the benefit and priority of U.S. nonprovisional patent application number 16 / 741,580, filed January 13, 2020, the disclosure of which is incorporated herein by reference in its entirety. Technical Field

[0003] This technology relates to distributed storage systems, and more particularly to the placement of master data in distributed storage systems. Background Technology

[0004] The proliferation of internet-enabled devices has generated enormous demand for internet services and content. In many ways, we have become an interconnected society, with users increasingly reliant on online services and content. This internet and interconnectivity revolution has presented significant challenges to content and storage providers, who often fall short of performance expectations while struggling to meet the massive influx of customer requests. For example, data providers typically require large and complex data centers to meet users' network and data demands. These data centers are usually equipped with server clusters configured to host specific data and services, and include numerous network devices configured to route and process data requests. In many cases, a particular data center is expected to handle millions of traffic and data requests.

[0005] Unsurprisingly, such large volumes of data can be difficult to manage, leading to significant performance degradation and challenges. In some cases, load balancing solutions can be implemented to improve performance and service reliability. However, current load balancing solutions are prone to node failures, often fail to adequately account for dynamic changes and fluctuations in network and data requests, and can be susceptible to latency and bottlenecks. Additional resources can be purchased and implemented to increase network capacity, thereby reducing latency and performance issues. Unfortunately, this approach is very expensive, increases network complexity, and is vulnerable to network fluctuations and changing data access patterns, potentially resulting in latency under overload conditions, waste under underload conditions, and highly variable performance. Attached Figure Description

[0006] To describe how the various advantages and features of this disclosure are obtained, the principles briefly described above will be described in more detail with reference to specific embodiments shown in the accompanying drawings. It should be understood that these drawings depict only exemplary embodiments of this disclosure and should not be considered as limiting its scope. The principles herein are described and explained with additional features and details using the drawings, in which:

[0007] Figure 1 An example distributed storage system is shown based on some examples;

[0008] Figure 2 The illustration shows a scenario for placing example data items in a distributed storage system, based on several examples.

[0009] Figure 3 The illustration shows a scenario for placing sample group data items in a distributed storage system, based on some examples.

[0010] Figure 4 This illustrates another example scenario of group data item placement in a distributed storage system, based on some examples.

[0011] Figure 5 The example methods for managing the placement of data items on a distributed storage system are shown, based on some examples.

[0012] Figure 6 Example network devices are shown based on some examples;

[0013] Figure 7 An example computing device is shown based on some examples. Detailed Implementation

[0014] Various embodiments of this disclosure are discussed in detail below. While specific implementations are discussed, it should be understood that this is for illustrative purposes. Those skilled in the art will recognize that other components and configurations can be used without departing from the spirit and scope of this disclosure. Additional features and advantages of this disclosure will be set forth in the following description and will be apparent in part from the description, or may be learned by practicing the principles disclosed herein. The features and advantages of this disclosure can be realized and obtained by means of the apparatuses and combinations particularly pointed out in the appended claims. These and other features of this disclosure will become more apparent from the following description and the appended claims, or may be learned by practicing the principles set forth herein.

[0015] Overview

[0016] This paper discloses systems, methods, and computer-readable media for intelligent and dynamic management of data item placement in distributed stateful storage systems. The data placement techniques described herein can reduce data access latency and improve data access performance by intelligently placing primary copies of data items at specific locations on a distributed storage system based on access patterns, network statistics, and / or events or conditions.

[0017] According to at least one example, a method for managing the placement of data items in a distributed storage system is provided. The method may include determining the current location of a primary copy of a data item stored on the distributed storage system, wherein the current location of the primary copy of the data item includes current data storage from a plurality of data storages on the distributed storage system; determining an access pattern associated with the primary copy of the data item, the access pattern including one or more starting locations of a set of access requests received by the distributed storage system for the primary copy of the data item, and a corresponding number of access requests received from each of the one or more starting locations; determining different locations on the distributed storage system for storing the primary copy of the data item based on the access pattern associated with the primary copy of the data item, the different locations including different data storages from the plurality of data storages; and placing the primary copy of the data item at the different locations on the distributed storage system.

[0018] According to at least one example, a system for managing the placement of data items in a distributed storage system is provided. The system may include one or more processors and at least one computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the system to determine the current location of a primary copy of a data item stored on the distributed storage system, wherein the current location of the primary copy of the data item includes current data storage from a plurality of data storages on the distributed storage system; determine an access pattern associated with the primary copy of the data item, the access pattern including one or more starting locations of a set of access requests to the primary copy of the data item received by the distributed storage system, and a corresponding number of access requests received from each of the one or more starting locations; based on the access pattern associated with the primary copy of the data item, determine different locations on the distributed storage system for storing the primary copy of the data item, the different locations including different data storages from the plurality of data storages; and place the primary copy of the data item at the different locations on the distributed storage system.

[0019] According to at least one example, a non-transitory computer-readable storage medium is provided for managing the placement of data items in a distributed storage system. The non-transitory computer-readable storage medium may store instructions that, when executed by one or more processors, cause the one or more processors to determine: a current location of a primary copy of a data item stored on the distributed storage system, wherein the current location of the primary copy of the data item includes current data storages from a plurality of data storages on the distributed storage system; determine an access pattern associated with the primary copy of the data item, the access pattern including one or more starting locations of a set of access requests for the primary copy of the data item received by the distributed storage system, and a corresponding number of access requests received from each of the one or more starting locations; determine different locations on the distributed storage system for storing the primary copy of the data item based on the access pattern associated with the primary copy of the data item, the different locations including different data storages from the plurality of data storages; and place the primary copy of the data item at the different locations on the distributed storage system.

[0020] In at least some aspects, the methods, systems, and non-transitory computer-readable storage media described above may include collecting information associated with a distributed storage system, including statistics associated with one or more resources, one or more data access restrictions associated with one or more data items on the distributed storage system, one or more events associated with at least one of the distributed storage system and one or more networks associated with the distributed storage system, data access patterns, and / or network statistics, wherein the one or more resources include storage nodes, compute nodes, virtual machines, software containers, servers, networks, and / or networking devices; based on the information associated with the distributed storage system, determining a data placement action estimated to improve data access performance associated with one or more data items on the distributed storage system or to improve the performance of the distributed storage system, the data placement action including moving at least one data item from at least one storage location to at least one different storage location, the at least one storage location and the at least one different storage location including different data stores from the plurality of data stores; and in response to determining the data placement action, moving the at least one data item from the at least one storage location to the at least one different storage location.

[0021] In some aspects, determining different locations on a distributed storage system for storing primary replicas of data items may include: identifying, from the one or more starting locations, the starting location of the highest number of access requests associated with the primary replica of the data item, based on the access patterns associated with the primary replica of the data item; determining which of the plurality of data stores is closest to the starting location of the highest number of access requests associated with the primary replica of the data item; and determining that the different data stores associated with the different locations are located closest to the starting location of the highest number of access requests associated with the primary replica of the data item.

[0022] In some examples, the one or more starting locations may correspond to one or more client devices that generate a set of access requests for a primary copy of a data item, which are received by the distributed storage system. Furthermore, in some examples, placing the primary copy of a data item at different locations on the distributed storage system may include moving the primary copy of the data item from the current data storage to a different data storage.

[0023] In some cases, determining that the different data storage devices are located at the starting position of the highest number of access requests associated with the primary copy of the data item may include determining that the number of hops between the different data storage devices and the starting position is less than the corresponding number of hops between each of the plurality of data storage devices and each of the one or more remaining positions from the one or more starting positions, and / or determining that the distance between the different data storage devices and the starting position is less than the corresponding distance between each of the plurality of data storage devices and each of the one or more remaining positions from the one or more starting positions.

[0024] In some aspects, determining different locations on a distributed storage system for storing primary copies of data items may include: determining a second current location of a second primary copy of a second data item stored on the distributed storage system, wherein the second current location of the second primary copy of the second data item includes a second current data storage from the plurality of data storages on the distributed storage system; selecting different locations on the distributed storage system for storing the primary copy of the second data item and the second primary copy of the second data item based on the access mode associated with the primary copy of the data item and the second access mode associated with the second primary copy of the second data item; and placing the primary copy of the data item and the second primary copy of the second data item at different locations on the distributed storage system.

[0025] In some examples, the second access pattern may include one or more corresponding starting locations for a second set of access requests associated with a second primary copy of the second data item, and a second corresponding number of access requests received from each of the one or more corresponding starting locations. In some cases, placing the primary copy of the data item and the second primary copy of the second data item at different locations on the distributed storage system may include moving the primary copy of the data item from its current location to a different location, and moving the second primary copy of the second data item from a second current location to a different location. Furthermore, in some examples, the primary copy of the data item may include a first partition of a partitioned dataset, and the second primary copy of the second data item may include a second partition of the partitioned dataset.

[0026] In some aspects, the methods, systems, and non-transitory computer-readable storage media described above may include determining that a primary copy of the data item and a second data item on the distributed storage system have been accessed together a threshold number of times; and after determining different locations on the distributed storage system for storing the primary copy of the data item, moving the second data item from its current corresponding location to the different location, the current corresponding location including one of the plurality of data storages on the distributed storage system.

[0027] The above-described method, system, and non-transitory computer-readable storage medium may include: determining that the primary copy of the data item includes a reference to a specific copy of a second data item stored on the distributed storage system, wherein the specific copy of the second data item includes one of a corresponding primary copy of the second data item or a copy of the corresponding primary copy of the second data item; selecting a different location on the distributed storage system for storing the specific copy of the second data item in response to determining a different location on the distributed storage system for storing the primary copy of the data item and determining that the primary copy of the data item includes the reference to the specific copy of the second data item; and moving the specific copy of the second data item from the corresponding location on the distributed storage system to the different location on the distributed storage system.

[0028] This summary is not intended to identify key or essential features of the claimed subject matter, nor is it intended to be used in isolation to determine the scope of the claimed subject matter. The subject matter should be understood by referring to the entire specification of this application, any or all of the accompanying drawings, and the appropriate portions of each claim.

[0029] The foregoing and other features and embodiments will become more apparent from the following description, claims and drawings.

[0030] describe

[0031] Distributed stateful storage systems manage state in a network environment to provide read and write access to data items. In some examples, state can be managed centrally, with one or more data stores residing in a central location (e.g., the core network) configured to manage read and write access to data items. Here, clients can perform read and write operations by sending requests to one or more data stores at the central location. In some cases, read-only state replicas can be managed in a decentralized manner. For example, read-only state replicas can be managed by one or more data stores located at the network "edge" (e.g., a cache). One or more data stores at the network edge can manage read-only access to data items, and clients can perform read operations by sending requests from one or more data stores at the network edge to the decentralized data store (e.g., a cache).

[0032] While implementing data storage or caching at the network edge can improve read operation performance, write operations will experience latency and performance degradation because requesting clients are typically located far from the central data storage. For example, the distance between a client and the centralized data storage that provides write access to data items to such a client increases the latency of write operations from such a client and degrades data access performance because the increased distance increases the number of networks, devices, and potential bottlenecks that data access communication between such a client and the centralized data storage traverses.

[0033] In some cases, to reduce latency from client write operations and improve data access performance, the method presented in this paper can intelligently place primary replicas of data items strategically within a distributed storage environment based on data access patterns, such as data access location (e.g., the client's location relative to the data item) and frequency. For example, based on access patterns calculated for the primary replicas of a particular client and / or data item, the primary replicas of data items frequently accessed by that client can be moved to a data store closer to that client. This can reduce the number of hops in data access communication to and from the client, reduce the number of potential bottlenecks encountered by such data access communication, and generally reduce latency and improve data access performance for that client.

[0034] In some cases, other factors may also be considered when selecting the location for storing or moving the primary copy of a data item. For example, when determining the strategic location for placing the primary copy (and / or duplicates) of a data item, the system may consider various factors such as cost considerations, network performance considerations, client access requirements, client subscription levels (e.g., advanced access level, standard access level, etc.), access patterns associated with a set of data items and / or clients, the type of data item, the relationship between the data item and other data items (e.g., partitioning of data items, referencing data items, etc.), historical and / or predicted access patterns, etc.

[0035] As further described below, the disclosed technology provides systems, methods, and computer-readable media for placing master data in distributed storage systems. This technology will be described in the following disclosure. The discussion begins as follows... Figure 1 The example distributed storage system shown is described and, as well as... Figures 2 to 4 This section describes various examples and techniques for placing master data in a distributed storage system. Example methods for placing master data in a distributed storage system are then described, such as... Figure 5 As shown. The discussion concludes with... Figure 6 The example network device shown and such Figure 7 The description shown includes example computing device architectures suitable for performing storage and computing operations. This disclosure now turns to... Figure 1 .

[0036] Figure 1 This is a simplified block diagram of an example distributed storage system 100 based on some examples. In this example, the distributed storage system 100 includes a core 102 and edges 110A-N (collectively referred to as "110"). The core 102 can act as the backbone, centralized network, and / or central hub for the network and storage services provided by the distributed storage system 100. Furthermore, the core 102 may include one or more networks (e.g., cloud networks, data centers, etc.) and / or one or more segments of the network environment of the distributed storage system 100, such as core segments or hubs.

[0037] Edge 110 may be connected to core 102 via one or more network devices (e.g., one or more switches or routers) and / or via one or more networks (e.g., one or more public networks (e.g., wide area networks, public clouds, etc.), one or more private networks (e.g., private data centers, private clouds, local area networks, virtual private networks, etc.), and / or one or more hybrid networks (e.g., hybrid clouds, etc.)). In some examples, edge 110 may interconnect with each other via core 102 and / or directly (or without core 102).

[0038] In some cases, edge 110 may represent a segment or portion of the network environment associated with distributed storage system 100. For example, edge 110 may be a segment or portion located at the edge or periphery of the network environment associated with core 102 and / or distributed storage system 100. In other cases, edge 110 may represent a separate network, such as a fog network, local area network (LAN), internal data center, enterprise network, etc. In some examples, such a network may be located at the edge or periphery of the network environment associated with distributed storage system 100. Therefore, edge 110 may be physically and / or logically closer to one or more clients 160-166 than core 102.

[0039] Core 102 may include one or more storage nodes 104 for storing or hosting one or more data stores 106. Similarly, edge 110 may include storage nodes 112 for storing or hosting data stores 114-120. For example, edge 110A may include one or more storage nodes 112 for storing one or more data stores 114, edge 110B may include one or more storage nodes 112 for storing one or more data stores 116, edge 110C may include one or more storage nodes 112 for storing one or more data stores 118, and edge 110N may include one or more storage nodes 112 for storing one or more data stores 120.

[0040] Storage nodes 104 and 112 may represent hardware and / or virtual storage infrastructure on distributed storage system 100. Furthermore, storage nodes 104 and 112 may include one or more physical storage servers, one or more virtual storage servers (e.g., virtual machines (VMs), software containers, etc.), one or more physical and / or logical storage components (e.g., storage drives, logical volumes, storage partitions, storage arrays, etc.), and / or any other physical and / or virtual / logical storage elements. Each of storage nodes 104 and 112 may be implemented by a single storage element, or may span or be distributed across multiple storage elements, providing a distributed storage infrastructure. In some cases, a storage node may span multiple physical or virtual storage elements. For example, a storage node may represent a virtual storage device, container, or location created from two or more physical servers and / or storage devices.

[0041] In some cases, storage nodes 104 and 112 may be grouped into storage node pools or clusters. For example, storage node 104 on core 102 may be grouped into one or more storage node pools or clusters, and storage node 112 on each of edge 110 may be grouped into one or more storage node pools or clusters. Storage nodes may be grouped into storage node pools or clusters based on one or more factors (e.g., one or more common characteristics). For example, storage nodes may be grouped into storage node pools or clusters according to storage type, data type (e.g., the data types they store), underlying storage platform, physical or virtual location, capacity, configuration settings or architecture, storage role, priority, network segment (e.g., IP prefix or subnet), shared resources, operating conditions, etc. In some cases, a pool or cluster of storage nodes (e.g., 104 and / or 112) may be configured to function as a single storage node or distributed storage. In other cases, a pool or cluster of storage nodes (e.g., 104 and / or 112) may represent a collection of storage nodes that can operate separately and / or individually.

[0042] Data storage device 106 on core 102 and data storage devices 114-120 on edge 110 may include libraries, containers, or structures for persistently storing and managing master data items 108A-N and replica data items 130A-N. Data storage device 106 on core 102 and data storage devices 114-120 on edge 110 may include, for example, databases, FDEs, file systems, storage systems, and / or any other data storage libraries. In some implementations, data storage devices 106 and 114-120 may include one or more databases.

[0043] Each of the primary data items 108A-N and replica data items 130A-N on data stores 106 and 114-120 may include, for example, a record (e.g., a database record), one or more database fields, a data object, a data structure containing data or data values, a collection of data elements, (multiple) data values, a data partition, a content item, etc. In some examples, data items on data stores 106 and / or 114-120 (e.g., 108A-N and / or 130A-N) may include any type of independent (or largely independent) data item, such as a configuration file, a record, a table row, a dataset that does not reference other data items (or references other data items to a limited extent), etc.

[0044] The primary data items 108A-N in data store 106 on core 102 can represent primary copies of replica data items 130A-N, while the replica data items 130A-N in data stores 114-120 on edge 110 can represent copies or read-only copies of the primary data items 108A-N in data store 106 on core 102. Primary data items 108A-N can be read-write data items and can provide the true source of replica data items 130A-N (e.g., current and / or authoritative data state and / or version). For example, primary data item 108A can be a read-write data item and can provide the current and authoritative state or version of the data associated with primary data item 108A and replica data item 130A, and can also provide the current and authoritative state or version of the data associated with primary data item 108N and replica data item 130N.

[0045] Clients 160-166 can access primary data items 108A-N and replica data items 130A-N through distributed storage system 100. Specifically, clients 160-166 can access primary data items 108A-N and replica data items 130A-N through core 102 and / or edge 110. For example, for read operations, clients 160-166 can access primary data items 108A-N and / or replica data items 130A-N from core 102 and / or edge 110, while for write operations, clients 160-166 can access primary data items 108A-N (e.g., the primary replica of replica data items 130A-N) through core 102. However, as further described below, primary data items 108A-N may be moved or placed at other locations on distributed storage system 100 based on one or more factors. Therefore, in some cases, clients 160-166 can access primary data items 108A-N from one or more of the edges 110.

[0046] Clients 160-166 can represent any computing device or network. For example, in some cases, clients 160-166 may include one or more client endpoints (e.g., client computing devices such as personal computers, smartphones, tablets, smart TVs, gaming systems, set-top boxes, smart wearables, etc.), one or more servers, one or more Internet of Things (IoT) devices, one or more autonomous vehicles, one or more network devices (e.g., switches, routers, etc.). In other cases, clients 160-166 may include one or more networks, such as one or more LANs, one or more data centers, one or more enterprise networks, one or more campus networks, one or more private networks, etc.

[0047] Distributed storage system 100 may include a coordinator system 140, which can collect and analyze information about distributed storage system 100 and coordinate or arrange the placement or movement of master data items 108A-N and replica data items 130A-N on distributed storage system 100. Furthermore, coordinator system 100 may include one or more computing devices (physical and / or virtual), or may be implemented by one or more computing devices. For example, in some cases, coordinator system 100 may be implemented by a server, network controller, coordinator device, router, or any other computing device. In other cases, coordinator system 100 may be implemented by multiple devices (e.g., multiple servers, multiple network controllers, multiple coordinator devices, multiple routers, etc.).

[0048] In some examples, the coordinator system 140 may track and monitor statistics and information associated with the distributed storage system 100, master data items 108A-N and / or replica data items 130A-N, and, based on the statistics and information tracked and monitored by the coordinator system 140, move (or instruct the distributed storage system 100 to move) one or more of the master data items 108A-N and / or replica data items 130A-N to specific locations(s) within the distributed storage system 100.

[0049] To illustrate, if statistics and information monitored by the coordinator system 140 indicate that client 160 frequently accesses primary data item 108A from the location closest to edge 110A, the coordinator system 140 can trigger a movement of primary data item 108A from data store 106 on core 102 to data store 114 on edge 110A. Such a movement reduces the distance (and hop count and / or potential bottleneck number) between primary data item 108A and the client 160 that frequently accesses primary data item 108A, thereby reducing latency and improving the access performance of client 160 when accessing primary data item 108A from data store 114 on edge 110A, rather than from data store 106 on core 102.

[0050] In some cases, the coordinator system 140 can use the collected information to coordinate or schedule the movement of replicated data items 130A-N on the distributed storage system 100. For example, as described in the previous example, when the primary data item 108A is moved from data store 106 on core 102 to data store 114 on edge 110A, the coordinator system 140 can also move the replicated data item 130A from data store 114 on edge 110A to data store 106 on core 102. As another example, if the information collected and monitored by the coordinator system 140 indicates that the replicated data item 130A is frequently accessed by client 164 from a location closest to core 102 (or where the response time received from core 102 is faster than from edge 110), the coordinator system 140 can trigger the movement of the replicated data item 130A from one of the edges 110 to core 102. As yet another example, if the information collected and monitored by the coordinator system 140 indicates that the replica data item 130N is rarely accessed from the edge 110N, the coordinator system 140 may trigger the movement of the replica data item 130N from the edge 110N to the core 102 to reduce network and / or resource utilization at the edge 110N (and thereby increase bandwidth, reduce congestion and increase resource availability at the edge 110N).

[0051] Furthermore, in some cases, the coordinator system 140 may designate a copy of the master data item stored in a different destination data store (e.g., replica data item 130A or 130N) as the master data item, and designate the previous master data item in the current data store as the replica data item, instead of moving the master data item from the current data store to a different destination data store. In some examples, if the coordinator system 140 changes the designation of the replica data item as the master data item, instead of moving the master data item to the location of the replica data item, the coordinator system 140 may perform consistency checks to ensure that the replica data item designated as the master data item reflects the latest data or is not outdated, and / or to check that designating the replica data item as the master data item does not produce any conflicts or errors.

[0052] In some cases, if the coordinator system 140 determines that designating a replica data item as the primary data item would result in a conflict or error, and / or the replica data item does not reflect the latest data or is outdated, the coordinator system 140 may attempt to correct such conflicts or errors and / or update the replica data item to reflect the latest data before designating it as the primary data item. In other cases, the coordinator system 140 may decide not to designate the replica data item as the primary data item, but instead decide to move the actual primary data item to the location where the replica data item is stored, rather than attempting to correct such conflicts or errors and / or update the replica data item to reflect the latest data. In such cases, the coordinator system 140 may leave the replica data item as a copy and may leave it in its current location or move it to a different location, such as the location where the primary data item was moved out.

[0053] In some examples, statistics and information tracked / monitored by the coordinator system 140 and used to trigger data placement actions may include access patterns and / or metrics associated with one or more of the primary data items 108A-N, replica data items 130A-N, core 102, edge 110, and / or one or more of the clients 160-166. In some cases, the coordinator system 140 may use such information to determine whether access latency and / or access performance associated with a particular primary data item and / or replica data item can be improved by moving the primary data item or replica data item to a different location in the distributed storage system 100 (e.g., different data stores, storage nodes, network segments, and / or networks). For example, the coordinator system 140 can use such information to determine whether moving the master data item or replica data item to the nearest specific data store and / or location (e.g., core 102, edge 110A, edge 110B, edge 110C, or edge 110A) to the client that interacts with the master data item or replica data item the most or most frequently (e.g., read and / or write access operations or interactions) would improve access statistics for the master data item or replica data item, such as access latency or delay, round-trip time (RTT) for access requests / operations, error rate, etc.

[0054] Non-limiting examples of access pattern information that the coordinator system 100 may use to determine / execute data item movement may include the locations (e.g., network address, physical location, network, geographic region, storage node, data store, etc.) where data items (e.g., 108A-N and / or 130A-N) on the distributed storage system 100 have been accessed over a period of time, and / or the typical locations (e.g., the locations of clients 160-166 that have accessed the data items), the access frequency of data items (e.g., 108A-N and / or 130A-N) from these locations (e.g., how many times and / or how often each data item was accessed from each location and / or by each client at each location), which clients (e.g., 160-166) and / or how many clients accessed the various data items (e.g., 108A-N and / or 130A-N). The types of access (e.g., read, write, etc.) from each access location and / or client access, which other data items (if any) were accessed in conjunction with or associated with a specific data item, which other operations were performed in conjunction with or associated with the access operations corresponding to the data items (e.g., 108A-N and / or 130A-N) and / or access locations, the time and / or date (and / or relative access frequency between different times and / or dates) of accessing the data items (e.g., 108A-N and / or 130A-N) from the access locations, latency and / or other performance or access metrics observed for requests / operations from the access locations, routing information (e.g., hop count, network path, routing cost, latency, performance, security, bandwidth, congestion, etc.) associated with access to the data items (e.g., 108A-N and / or 130A-N) and / or access locations, and / or any other access pattern information.

[0055] In some cases, the information tracked / monitored by the coordinator system 140 and used to trigger data placement actions may include other types of information that can affect the status, security, stability, and / or performance of the distributed storage system 100, and / or the performance, cost, and / or other metrics of access requests or operations received or processed by the distributed storage system 100. For example, this information may include the location of master data items 108A-N and replica data items 130A-N, the status / condition of one or more components of the distributed storage system 100 (e.g., core 102, edge 110, storage nodes 104 and 112, data store 106, data stores 114-120, etc.), the response time of access requests associated with master data items 108A-N and / or replica data items 130A-N, the available bandwidth at one or more of core 102 and / or edge 110, and the resource usage and / or workload at one or more of core 102 and / or edge 110. Status, resource availability and / or processing capacity at one or more of cores 102 and / or edges 110, location of cores 102 and / or edges 110, topology of cores 102 and / or edges 110 associated with cores 102 and / or edges 110, performance metrics / statistics (e.g., input / output operations per second (IOPS), latency, bandwidth, throughput, packet loss, jitter, connectivity, retransmission, error rate, RTT, congestion, availability, utilization, etc.), client subscription levels associated with clients 160-166 (e.g., premium access, standard or basic access, guest access, etc.), etc.

[0056] In some cases, the coordinator system 100 may use information collected from or for the distributed storage system 100 to determine a data item placement scheme or action for placing or distributing one or more of the primary data items 108A-N and / or replica data items 130A-N at specific locations (e.g., data stores, storage nodes, segments, networks, etc.) on the distributed storage system 100. For example, the coordinator system 100 may use such information to determine a data item placement scheme estimated to better balance or distribute the load or burden on the distributed storage system 100, improve access performance to the primary data items 108A-N and / or replica data items 130A-N, reduce error rates, meet certain access criteria (e.g., access performance levels, quality of service (QoS) requirements, service level agreements (SLAs), security requirements, access restrictions, etc.), comply with data sovereignty laws / policies, improve resource availability, reduce resource overuse, reduce congestion, reduce costs, increase bandwidth, improve efficiency, reduce network traffic, etc.

[0057] To illustrate, the coordinator system 100 can use this information to determine the source and location of write requests associated with master data items 108A-N, and the corresponding frequency of write requests from each location, to identify a target location for moving or placing one or more of the master data items 108A-N. This target location can be, for example, a location on the distributed storage system 100 (e.g., edge, core, data store, storage node, etc.) that is closest to the location where the most frequent write requests originate. Moving or placing one or more of the master data items 108A-N at such a location reduces the distance traveled by write requests to such master data item(s) from that location (which is the most frequently accessed location and therefore accounts for the majority of write requests to such master data item(s)) to the location on the distributed storage system 100 where such master data item(s) ...

[0058] The coordinator system 140 can receive information collected and monitored / tracked by the distributed storage system 100 (e.g., core 102, edge 110, one or more network devices on the distributed storage system 100, etc.) on a push and / or pull basis. This information includes data item access history and / or statistics, operating conditions at different locations on the distributed storage system 100, data item information, topology information associated with the distributed storage system 100, performance metrics, storage or placement information, resource parameters, request parameters, configuration information, etc. For example, the coordinator system 140 can pull corresponding data streams from the distributed storage system 100, such as access patterns, statistics, metrics, status information, operating conditions, etc. As another example, the distributed storage system 100 can push such corresponding data streams to the coordinator system 140.

[0059] Figure 1 The elements and components shown are illustrative examples provided for purposes of explanation. Therefore, although... Figure 1 The illustration shows a certain type and number of networks or segments (e.g., core 102, edge 110A to 110N) and components (e.g., storage nodes 104 and 112, data stores 106 and 114-120, master data items 108A-N, replica data items 130A-N, etc.). Those skilled in the art will recognize that other examples may include […]. Figure 1 The different numbers and / or types of networks / segments and / or components shown. For example, in some cases, distributed storage system 100 may include more than Figure 1The more or fewer edges, cores or core segments and / or components shown (e.g., storage nodes 104 and 112, data stores 106 and 114-120, master data items 108A-N, replica data items 130A-N, devices, etc.).

[0060] Figure 2 An example data item placement scenario 200 in a distributed storage system 100 is illustrated. In this example, client 160 is closer to / most close to edge 110A, client 162 is closer to / most close to edge 110B, client 164 is closer to / most close to edge 110C, and client 166 is closer to / most close to edge 110N. Furthermore, a coordinator system 140 can receive and monitor the placement data 202 from the distributed storage system 100.

[0061] Placement data 202 may include metrics, statistics, event data, status information, notifications, configuration information, job / workload information, data item information, access pattern information, parameters, preferences, client information, access and / or data restriction information, location information, resource information, network information, etc. In some examples, placement data 202 may include a history of access requests and / or operations (e.g., access patterns) associated with the master data items 108A-N and the replica data items 130A-N. The history of access requests and / or operations can indicate one or more access patterns associated with the master data items 108A-N and the replica data items 130A-N, such as which clients 160-166 have accessed the master data items 108A-N and the replica data items 130A-N, the location of the access requests and / or operations associated with the master data items 108A-N and the replica data items 130A-N (e.g., the location of clients 160-166 associated with the access requests and / or operations), the frequency of access requests and / or operations from the location associated with the access requests and / or operations, the type of access requests and / or operations (e.g., read, write, etc.), the date / time of such access requests and / or operations, performance statistics associated with the access requests and / or operations, access anomalies, access trends, etc.

[0062] For example, placement data 202 can indicate that client 160 is closest to edge 110A (relative to edges 110B-N and core 102), client 162 is closest to edge 110B (relative to edges 110A, edges 110C-N and core 102), client 164 is closest to edge 110C (relative to edges 110A-B, edges 110N and core 102), and client 166 is closest to edge 110N (relative to edges 110A-C and core 102). Placement data 202 can also indicate that client 162 has accessed master data item 108A on data storage 106 of core 102 more frequently and / or more times than clients 160 and 164-166, and that client 166 has accessed master data item 108N on data storage 106 of core 102 more frequently and / or more times than clients 160-164. As further explained below, the coordinator system 140 can use this information in the placement data 202 to move the master data items 108A and 108N closer to the clients 162 and 166, respectively, thereby reducing latency and improving the performance of access requests and operations to the master data items 108A and 108N from the clients 162 and 166.

[0063] In some examples, placing data 202 may include information about core 102, edge 110 and / or components of core 102 and / or edge 110 (e.g., storage nodes 104 and 112, data store 106 on core 102, data stores 114-120 on edge 110, master data items 108A-N and replica data items 130A-N, etc.) (e.g., performance statistics, bandwidth, congestion, capacity, latency, resource availability, status, conditions, metrics, location information, configuration information, access restrictions, access policies, etc.)).

[0064] The coordinator system 140 can analyze the placement data 202 and, based on the access patterns of clients 160-166 and / or any other information associated with the distributed storage system 100, determine whether any of the primary data items 108A-N and / or replica data items 130A-N should be moved (e.g., whether moving such data items would produce one or more improvements / benefits). In other words, the coordinator system 140 can analyze the placement data 202 and determine whether moving any of the primary data items 108A-N and / or replica data items 130A-N to a different storage location would reduce latency of future access requests / operations, improve the performance of future access requests / operations, reduce load / burden on the distributed storage system 100, improve resource utilization at the distributed storage system 100, reduce congestion, optimize bandwidth usage / availability, improve resource availability at the distributed storage system 100, and / or provide any other benefits for data access requests / operations and / or network / system conditions.

[0065] For example, as previously described, placement data 202 can indicate that client 162 (relative to clients 160 and 164-166) accesses master data item 108A on core 102 most frequently. Therefore, to improve the access performance and efficiency of future requests from client 162 to master data item 108A and from client 166 to master data item 108N, coordinator system 140 can move master data items 108A and 108N closer to clients 162 and 166, thereby reducing latency and improving the performance of future access requests from clients 162 and 166 to master data items 108A and 108N.

[0066] Therefore, in some examples, after determining (e.g., based on placement data 202) that client 162 accesses the master data item 108A on core 102 most frequently, and that client 162 is closer to / closest to edge 110B than other storage locations on the distributed storage system 100 (e.g., core 102 and edges 110A, 110C, and 110N), the coordinator system 140 may decide to move the master data item 108A from core 102 (e.g., from data store 106) to edge 110B (e.g., to data store 116) to improve performance and reduce latency for requests / operations on the master data item 108A from client 162.

[0067] Similarly, after determining (e.g., based on placement data 202) that client 166 most frequently accesses the master data item 108N on core 102, and that client 166 is closer to / closest to edge 110N than other storage locations on distributed storage system 100 (e.g., core 102 and edges 110A, 110B, and 110C), coordinator system 140 may decide to move the master data item 108N from core 102 (e.g., from data store 106) to edge 110N (e.g., to data store 120) to improve performance and reduce latency of requests / operations on master data item 108N from client 166.

[0068] To move data items or trigger placement actions, the coordinator system 140 may send a placement request 204 to the distributed storage system 100 (e.g., to core 102, edge 110A, edge 110B, edge 110C, edge 110N, and / or network devices within the distributed storage system 100). For example, the coordinator system 140 may send a placement request 204 to core 102, which may trigger the movement of primary data item 108A to edge 110B and the movement of primary data item 108N to edge 110N. The placement request 204 may include, for example, commands, instructions, and / or requests to move primary data item 108A to edge 110B and primary data item 108N to edge 110N.

[0069] Distributed storage system 100 may receive placement requests 204 from coordinator system 140 (e.g., at core 102) and execute data placement actions 206 and 208 based on or triggered by the requests in placement request 204. In some examples, data placement action 206 may be a movement of primary data item 108A from core 102 to edge 110B, thus storing primary data item 108A closer to the client 162 that most frequently accesses primary data item 108A. Similarly, in some examples, data placement action 208 may be a movement of primary data item 108N from core 102 to edge 110N, thus storing primary data item 108N closer to the client 166 that most frequently accesses primary data item 108N.

[0070] Following data placement actions 206 and 208, primary data item 108A will be hosted by edge 110B instead of core 102 (e.g., stored on edge 110B), and primary data item 108N will be hosted by edge 110N instead of core 102 (e.g., stored on edge 110N). Therefore, in future instances, client 162 will be able to access primary data item 108A from the corresponding data store 116 at edge 110B, and client 166 will be able to access primary data item 108N from the corresponding data store 120 at edge 110N. By placing primary data item 108A on edge 110B, coordinator system 140 can reduce the distance between client 162 and primary data item 108A, thereby reducing latency and improving the performance of future access requests and operations for primary data item 108A by client 162. Furthermore, by placing the master data item 108N on the edge 110N, the coordinator system 140 can reduce the distance between the client 166 and the master data item 108N, thereby reducing latency and improving the performance of future access requests and operations of the client 166 to the master data item 108N.

[0071] The coordinator system 140 can continue to receive and monitor placement data 202 to determine whether additional data placement actions should be performed. For example, if new placement data indicates that client 160 is now the client that most frequently accesses (reads and / or writes) the master data item 108A previously moved to edge 110B, the coordinator system 140 can send a new placement request to the distributed storage system 100 to trigger the movement of master data item 108A from edge 110B to edge 110A closer to client 160.

[0072] While in the examples above, data placement actions 206 and 208 are described as involving moving master data items 108A and 108N to locations closest to the clients 162 and 166 that access master data items 108A and 108N most frequently (e.g., edge 110B and 110N and data stores 116 and 120), it should be noted that in other examples, data placement actions 206 and 208 may move master data items 108A and 108N to other locations (e.g., other data stores and / or edge, core, etc.) and / or based on other factors and / or access patterns.

[0073] For example, in some cases, it may be based on one or more conditions of the source and / or destination location (e.g., data storage, core, edge, etc.) (e.g., bandwidth, resource availability, congestion, connectivity or downtime, error rate, status, performance, access restrictions, etc.), one or more characteristics of the source and / or destination location (e.g., data storage, core, edge, etc.) (e.g., location, platform or infrastructure, configuration, performance statistics, relative ranking, network type, data type, resource type, etc.), or one or more events (e.g., traffic fluctuations, one or more networks or resources). Failure, one or more errors, one or more network changes, one or more preferences or requirements, cost and / or client subscription level, other access patterns (e.g., read and / or write access patterns associated with a set of data items, clients and / or locations), and / or any other factor or combination of factors that move master data items (e.g., 108A, 108B or 108N) to or out of a specific location (e.g., data stores 106, 114, 116, 118 or 120; core 102; edge 110A, 110B, 110C or edge 110N, etc.).

[0074] Figure 3 An example group data item placement scenario 300 in a distributed storage system 100 is illustrated. As previously described, a coordinator system 140 can receive and monitor placement data 202 from the distributed storage system 100. In this example, placement data 202 may include access patterns associated with partition data 304. Partition data 304 may include master data items 108A-N. For example, each of the master data items 108A-N may represent a partition of a partitioned dataset (e.g., 304), and the master data items 108A-N together may constitute partition data 304. Therefore, in this example, placement data 202 may provide access patterns for different data partitions (e.g., 108A-N) in the partitioned dataset (e.g., 304).

[0075] In some cases, placement data 202 may include statistics used to identify the corresponding location of clients accessing partition data 304 and / or each individual partition (e.g., primary data items 108A-N) within partition data 304, and the corresponding access frequency of these clients. In some examples, placement data 202 may also include statistics identifying access patterns of other data items such as replica data items 130A-N. Coordinator system 140 can therefore analyze placement data 202 and determine that clients 160 and 162 access partition data 304 most frequently. In some cases, coordinator system 140 may also determine (e.g., based on placement data 202) that client 160 is closest to edge 110A, and client 162 is closest to edge 110B. Coordinator system 140 may also determine that, although client 160 is closest to edge 110A, edge 110B is also closer to client 160 than core 102 (and edges 110C and 110N).

[0076] Based on this information, the coordinator system 140 can determine that moving partition data 304, including master data items 108A-N, to edge 110B can provide the greatest improvement and / or performance and / or cost advantage (and / or a balance of advantages and disadvantages) for access requests / operations associated with partition data 304 as a whole and / or one or more of its components (e.g., master data items 108A-N). Therefore, after determining that moving partition data 304 to edge 110B can provide the greatest improvement and / or performance and / or cost advantage (and / or the best balance of advantages and disadvantages) for access requests / operations associated with partition data 304, the coordinator system 140 can send a group placement request 302 to distributed storage system 100 (e.g., to core 102, edge 110B, and / or network devices in distributed storage system 100).

[0077] The group placement request 302 may include an instruction to move partition data 304, including master data items 108A-N, to edge 110B. This instruction may then trigger a group placement action 304, which may include moving partition data 304, including master data items 108A-N, to edge 110B (e.g., to one or more of data stores 116 on edge 110B). After the group placement action 304, partition data 304 will be hosted by edge 110B instead of core 102 (e.g., stored on edge 110B). Therefore, in the future, clients 160 and 162 (and any other clients) will be able to access partition data 304 (and / or any master data items 108A-N in partition data 304) from edge 110B, which is closer to clients 160 and 162 than core 102, thus potentially resulting in improved access performance (e.g., reduced latency), cost improvements (benefits), and / or other improvements (benefits). For example, by placing partition data 304 on edge 110B, coordinator system 140 can reduce the distance between clients 160 and 162 and partition data 304, thereby reducing latency and improving the performance of future access requests and operations of clients 160 and 162 to partition data 304.

[0078] The coordinator system 140 can continue to receive and monitor placement data 202 to determine whether additional placement actions should be performed. For example, if new placement data indicates that clients 164 and / or 166 have accessed partition data 304 at a higher frequency than clients 160 and / or 162 within a configurable amount of time, the coordinator system 140 can determine whether partition data 304 should be moved to a different location. To illustrate, if the coordinator system 140 determines that clients 164 and 166 are closer to edge 110C than edge 110B, the coordinator system 140 can estimate whether moving partition data 304 from edge 110B to edge 110C would result in performance or other (e.g., cost, resource availability, congestion / bandwidth, etc.) improvements (overall and / or generally).

[0079] Coordinator system 140 may estimate whether moving partition data 304 from edge 110B to edge 110C would result in such an improvement based on one or more factors. For example, in some cases, coordinator system 140 may estimate whether moving partition data 304 from edge 110B to edge 110C would result in such an improvement based on the relative distance of client 160-166 to edges 110B and 110C, the frequency (and / or frequency difference) of client 160-166 accessing partition data 304, the date / time of client 160-166 accessing partition data 304, the bandwidth and / or congestion metrics associated with client 160-166 and / or 110C, the QoS requirements and / or SLA associated with client 160-166, and / or any other relevant factors.

[0080] As mentioned earlier, in Figure 3 In the example shown, client 160 is closest to edge 110A, and client 162 is closest to edge 110B. Although client 160 is closer to edge 110A, partition data 304 has been moved to edge 110B based on the access frequency of clients 160 and 162. Here, coordinator system 140 can determine that moving partition data 304 to edge 110B will provide a greater advantage / benefit (overall and / or generally) than moving partition data 304 to edge 110A, such as a greater performance improvement. For example, coordinator system 140 can compare the advantages and disadvantages of moving partition data 304 to edge 110A and edge 110B, and select edge 110B as the target location for partition data 304 based on the comparison of advantages and disadvantages. The coordinator system 140 can determine advantages and disadvantages based on, for example, the relative distances of clients 160 and 162 to edges 110A and 110B, the relative frequency of clients 160 and 162 accessing partition data 304, the types of operations performed by clients 160 and 162 on partition data 304 (e.g., read and / or write), the bandwidth and / or congestion metrics associated with clients 160 and 162 and / or edges 110A and 110B, the QoS requirements and / or SLAs associated with clients 160 and 162, and / or any other relevant factors.

[0081] Figure 4Another example group data item placement scenario 400 in a distributed storage system 100 is illustrated. As previously described, the coordinator system 140 can receive and monitor placement data 202 from the distributed storage system 100. In this example, placement data 202 may include access patterns associated with groups of data items, such as groups of data items from primary data items 108A-N and replica data items 130A-N. For example, placement data 202 may include statistics identifying one or more groups of data items that are frequently accessed together, as well as the corresponding locations of clients accessing one or more groups of data items and the corresponding access frequencies of these clients.

[0082] For illustration, placement data 202 may indicate that primary data items 108A and 108N are frequently accessed together by client 162 for write operations, and that client 162 is closer to edge 110B than core 102 and edges 110A, 110C, and 110N. Placement data 202 may indicate that client 162 (e.g., together and / or individually) accesses primary data items 108A and 108N more frequently than clients 160, 164, and 166 (e.g., together and / or individually). Based on this information, coordinator system 140 may determine that moving primary data items 108A and 108N to edge 110B will provide better performance and / or other benefits / improvements (overall and / or generally) than leaving primary data items 108A and 108N in their respective locations (e.g., core 102 and edge 110N) or moving primary data items 108A and 108N to different locations within the distributed storage system 100.

[0083] Therefore, after determining that moving master data items 108A and 108N to edge 110B can provide greater performance and / or benefits / advantages (and / or the best balance of advantages and disadvantages) for access requests / operations associated with master data items 108A and 108N, coordinator system 140 may send group placement request 402 to distributed storage system 100 (e.g., to core 102, edge 110B, edge 110N and / or network devices in distributed storage system 100).

[0084] Group placement request 402 may include an instruction to move primary data items 108A and 108N to edge 110B. This instruction may trigger group placement action 404, which may include moving primary data items 108A and 108N from core 102 to edge 110B (e.g., to one or more of data stores 116 on edge 110B). After group placement action 404, primary data item 108A will be hosted by edge 110B instead of core 102 (e.g., stored on edge 110B), and primary data item 108N will be hosted by edge 110B instead of edge 110N (e.g., stored on edge 110B). Therefore, in the future, client 162 (and any other client) will be able to access primary data items 108A and 108N from edge 110B, which is closest to client 162, and thus access performance improvements / benefits (e.g., reduced latency), cost improvements / benefits, and / or other improvements / benefits may occur. For example, by placing master data items 108A and 108N on edge 110B, coordinator system 140 can reduce the distance between client 162 and master data items 108A and 108N, thereby reducing latency and improving the performance of future access requests and operations from client 162 to master data items 108A and 108N. Overall and / or generally, such improvements / benefits can be estimated to outweigh any disadvantages of moving master data items 108A and 108N to edge 110B, such as any reduction in the performance of access requests / operations from clients 160, 164, and / or 166.

[0085] The coordinator system 140 can continue to receive and monitor placement data 202 to determine whether additional placement actions should be performed. If new placement data indicates that other clients(s) located closer to a different location in the distributed storage system 100 have accessed master data items 108A and / or 108N at a higher frequency than client 162 within a configurable time period, the coordinator system 140 can determine whether master data items 108A and / or 108N should be moved to that different location, as previously described.

[0086] Having disclosed the example system, components, and concepts, this disclosure now turns to an example method 500 for managing the placement of data items (e.g., 108A-N, 130A-N) on a distributed storage system (e.g., 100), such as... Figure 5 The steps outlined herein are non-limiting examples provided for illustrative purposes and can be implemented in any combination thereof, including excluding, adding, or modifying certain combinations of steps.

[0087] In step 502, method 500 may include determining the current location (e.g., core 102; edge 110A, 110B, 110C, or 110N; data store 106, 114, 116, 118, or 120) of the primary replica of a data item (e.g., primary data item 108A, 108B, or 108N) stored on a distributed storage system (e.g., 100). In some examples, the current location of the primary replica of the data item may include the current network or segment on the distributed storage system (e.g., core 102, edge 110A, edge 110B, edge 110C, or edge 110N) and / or the current data store (e.g., data stores 106, 114, 116, 118, or 120) from multiple data stores (e.g., data stores 106 and 114-120) on the distributed storage system.

[0088] In step 504, method 500 may include determining an access pattern associated with the primary copy of the data item. In some cases, the access pattern associated with the primary copy of the data item may include one or more starting positions of a set of access requests (e.g., read and / or write requests) to the primary copy of the data item received by the distributed storage system, the number of access requests received from each of the one or more starting positions, the type of access request (e.g., read, write, etc.), the date / time of the access request, data access trends (e.g., time-based data access trends, client-based data access trends, location-based access trends, access trends associated with a specific data item or type, access trends associated with a specific access trigger, access trends associated with a specific group of data items, data access sequences, access trends associated with a specific event, access trends associated with a specific condition, etc.), and / or any other access pattern characteristics.

[0089] In some examples, the access pattern associated with the primary replica of a data item includes one or more originating locations for a set of access requests (e.g., read and / or write requests) to the primary replica of the data item received by the distributed storage system, and multiple access requests received from each of the one or more originating locations. The one or more originating locations can refer to the location (e.g., network, client, address, geographic location, region, source, etc.) where the set of access requests originates (e.g., where the access requests are generated, transmitted, etc.). The number of access requests can refer to the quantity and / or frequency of access requests. For example, the number of access requests received from an originating location can refer to how many access requests originate from that location and / or the frequency with which access requests are received from that location.

[0090] In some cases, access patterns associated with the primary copy of a data item can be determined based on placement data collected and / or monitored from the distributed storage system (e.g., 202). For example, in some cases, the coordinator system (e.g., 140) can collect information from the distributed storage system such as access pattern statistics (e.g., data access requests or operations, the number and / or frequency of data access requests or operations, the location of the clients associated with the data access requests or operations, the location of the data items accessed by these clients from these locations, the distance between the locations of these data items and the locations of these clients, the date / time of these data access requests or operations, etc.), network statistics, resource statistics, data access restrictions, data access subscription levels, network and / or distributed storage system characteristics, system events, errors, failures, metrics, data item characteristics, client and / or data preferences, data types, network or system configuration information, data access policies, etc.).

[0091] In step 506, method 500 may include determining different locations on the distributed storage system (e.g., a different core 102; edge 110A, 110B, 110C, or 110N; data store 106, 114, 116, 118, or 120) for storing the primary copy of the data item based on the access patterns associated with the primary copy of the data item. In some examples, different locations may include different networks or segments on the distributed storage system (e.g., core 102, edge 110A, edge 110B, edge 110C, or edge 110N) and / or different data stores from multiple data stores.

[0092] In some aspects, determining different locations on a distributed storage system for storing primary replicas of data items may include: identifying, from the one or more starting locations, the starting location of the highest number of access requests associated with the primary replica of the data item, based on the access patterns associated with the primary replica of the data item; determining which of the plurality of data stores is closest (e.g., geographically closest, logically closest, etc.) to the starting location of the highest number of access requests associated with the primary replica of the data item; and determining that the different data stores associated with the different locations are located closest to the starting location of the highest number of access requests associated with the primary replica of the data item.

[0093] The starting point for the maximum number of access requests can refer to the location where the maximum number and / or frequency of access requests to the primary replica of the data item are generated, and / or the location where the distributed storage system receives the maximum number and / or frequency of access requests to the primary replica of the data item. In some examples, one or more starting points can correspond to one or more client devices (e.g., 160, 162, 164, 166) that generate a set of access requests to the primary replica of the data item that are received by the distributed storage system.

[0094] In some examples, determining that different data stores are located at the starting position of the highest number of access requests associated with the primary copy of the data item may include determining (e.g., associated with different positions) the number of hops between the different data stores and the corresponding number of hops between the starting position and each of the multiple data stores and each of the one or more remaining positions from the one or more starting positions (e.g., one or more starting positions other than the starting position of the highest number of access requests), and / or determining that the distance between the different data stores and the starting position is less than the corresponding distance between each of the multiple data stores and each of the one or more remaining positions from the one or more starting positions.

[0095] In other examples, determining that different data stores are located at the starting location of the highest number of access requests associated with the primary copy of the data item may alternatively or additionally include determining that the number of hops between the different data stores and the starting location is less than the corresponding number of hops between each of the multiple networks or segments (e.g., core 102, edge 110) associated with the distributed storage system and each of one or more remaining locations, and / or determining that the distance between the different data stores and the starting location is less than the corresponding distance between each of the multiple networks or segments (e.g., core 102, edge 110) associated with the distributed storage system and each of one or more remaining locations from one or more starting locations.

[0096] In some aspects, determining different locations on a distributed storage system for storing primary copies of data items may include determining a second current location of a second primary copy of a second data item (e.g., primary data item 108A, 108B, or 108N) stored on the distributed storage system; selecting different locations on the distributed storage system for storing the primary copy of the second data item and the second primary copy of the second data item based on the access mode associated with the primary copy of the data item and the second access mode associated with the second primary copy of the second data item; and placing the primary copy of the data item and the second primary copy of the second data item at different locations on the distributed storage system. In some examples, the second current location of the second primary copy of the second data item may include a second current data storage from multiple data storages on the distributed storage system. Furthermore, in some examples, the primary copy of the data item may include a first partition of a partitioned dataset (e.g., partitioned data 304), and the second primary copy of the second data item may include a second partition of the partitioned dataset.

[0097] In some cases, a second access pattern may include one or more corresponding starting locations (e.g., the starting or receiving location of the second set of access requests) associated with a second primary copy of the second data item, and a second corresponding number of access requests received from each of the one or more corresponding starting locations (e.g., the number and / or frequency of access requests). In some examples, placing the primary copy of the data item and the second primary copy of the data item in different locations on a distributed storage system may include moving or migrating the primary copy of the data item from its current location to a different location, and moving or migrating the second primary copy of the second data item from its second current location to a different location.

[0098] In step 508, method 500 may include placing a primary copy of the data item at different locations on the distributed storage system. In some examples, placing a primary copy of the data item at different locations on the distributed storage system may include moving or migrating a primary copy of the data item from the current data store to a different data store.

[0099] In some aspects, method 500 may include determining a threshold number of times and / or frequency that a primary copy of a data item and a second data item on a distributed storage system have been accessed together; and after determining the different locations on the distributed storage system used to store the primary copy of the data item, moving the second data item from its current corresponding location to the different location. The current corresponding location may, for example, be a data store among multiple data stores on the distributed storage system.

[0100] In some aspects, method 500 may include determining that the primary copy of a data item includes a reference (e.g., a pointer, association, link, relationship, etc.) to a specific copy of a second data item stored on a distributed storage system; selecting a different location on the distributed storage system for storing the specific copy of the second data item in response to determining that the primary copy of the data item includes the reference to the specific copy of the second data item; and moving the specific copy of the second data item from the corresponding location on the distributed storage system to the different location on the distributed storage system. In some examples, the specific copy of the second data item may include the primary copy of the second data item or a copy of the primary copy of the second data item.

[0101] In some implementations, method 500 may determine different locations for placing (e.g., moving, migrating, storing, etc.) data items based on access pattern information and / or one or more other factors. For example, in some aspects, method 500 may include collecting information associated with a distributed storage system; determining, based on the information associated with the distributed storage system, a data placement action estimated to improve data access performance (e.g., latency, response time, error rate, etc.) associated with one or more data items on the distributed storage system and / or improve the performance of the distributed storage system (e.g., the performance of the distributed storage system as a whole and / or the performance of one or more components / resources of the distributed storage system); and, in response to determining the data placement action, moving at least one data item (e.g., primary data items 108A, 108B, and / or 108N, and / or replica data items 130A and / or 130N) from at least one storage location to at least one different storage location (e.g., different data stores, core 102, edge 110A, edge 110B, edge 110C, or edge 110N).

[0102] In some examples, the information collected in association with the distributed storage system may include statistics associated with one or more resources (e.g., storage nodes 104 and / or 112, compute nodes, core 102, edge 110, network devices, bandwidth, etc.), one or more data access restrictions associated with one or more data items on the distributed storage system (e.g., location-based restrictions, policy-based restrictions, client-based restrictions, network-based restrictions, resource-based restrictions, source-based restrictions, data type-based restrictions, subscription-based restrictions, etc.), data storage restrictions (e.g., restrictions based on data sovereignty laws, data types, resource types, network utilization, resource utilization, date / time, resource availability, etc.), one or more events (e.g., errors, failures, network changes, traffic changes, access events, system events, etc.), data access patterns, and / or network statistics associated with one or more networks associated with the distributed storage system. In some examples, one or more resources may include storage nodes, compute nodes, virtual machines, software containers, servers, network and / or networking devices (e.g., switches, routers, firewalls, devices, etc.).

[0103] In some cases, data placement may include moving at least one data item from at least one storage location (e.g., data store, core 102, edge 110A, edge 110B, edge 110C, or edge 110N) to at least one different storage location. In some examples, at least one storage location and at least one different storage location may include different data stores from multiple data stores, different networks associated with a distributed storage system, different network segments, etc.

[0104] In some examples, when determining a data placement action (e.g., moving or migrating data items) based on multiple other factors (e.g., access patterns, performance statistics, resource metrics, environmental conditions, events, etc.), weights can be assigned to the factors used to determine the data placement action. For example, suppose the data placement action for a data item is determined based on the access frequency of data items from different access locations, the status of different storage nodes on the distributed storage system, and the amount of available bandwidth at different networks or segments on the distributed storage system. Here, different weights can be determined and applied to the access frequency, the status of different storage nodes, and the amount of available bandwidth used to determine the placement action. This allows the placement action to consider various factors, with some factors receiving higher weight, emphasis, or priority.

[0105] This disclosure now turns to Figure 6 and Figure 7 , Figure 6 and Figure 7Example network and computing devices are shown, such as switches, routers, nodes, servers, client devices, coordinators (e.g., coordinator system 140), etc.

[0106] Figure 6 An example network device 600 suitable for performing switching, routing, load balancing, and other networking operations is shown. Network device 600 includes a central processing unit (CPU) 604, an interface 602, and a bus 610 (e.g., a PCI bus). When operating under the control of appropriate software or firmware, CPU 604 is responsible for performing packet management, error detection, and / or routing functions. CPU 604 preferably performs all these functions under software control, including an operating system and any suitable application software. CPU 604 may include one or more processors 608, such as processors from the Intel x86 family of microprocessors. In some cases, processor 608 may be hardware specifically designed to control the operation of network device 600. In some cases, memory 606 (e.g., non-volatile RAM, ROM, etc.) also forms part of CPU 604. However, there are many different ways to couple memory to the system.

[0107] Interface 602 is typically provided as a modular interface card (sometimes called a "line card"). Typically, they control the sending and receiving of data packets on the network and sometimes support other peripheral devices used with network device 600. Available interfaces include Ethernet interfaces, Frame Relay interfaces, cable interfaces, DSL interfaces, token ring interfaces, etc. In addition, various high-speed interfaces can be provided, such as Fast Token Ring interfaces, wireless interfaces, Ethernet interfaces, Gigabit Ethernet interfaces, ATM interfaces, HSSI interfaces, POS interfaces, FDDI interfaces, WIFI interfaces, 3G / 4G / 5G cellular interfaces, CAN bus, LoRA, etc. Typically, these interfaces may include ports suitable for communication with the appropriate media. In some cases, they may also include a dedicated processor, and in some instances, volatile RAM. The dedicated processor can control communication-intensive tasks such as packet switching, media control, signal processing, cryptographic processing, and management. By providing a separate processor for communication-intensive tasks, these interfaces allow the main CPU (e.g., 604) to efficiently perform routing calculations, network diagnostics, security functions, etc.

[0108] Although Figure 6 The system shown is a specific network device of this disclosure, but is by no means the only network device architecture that can implement this disclosure. For example, an architecture with a single processor for handling communication and routing calculations is typically used. Furthermore, network device 600 may also use other types of interfaces and media.

[0109] Regardless of the network device's configuration, it may employ one or more memories or memory modules (including memory 606) configured to store program instructions for general network operations and mechanisms for the roaming, routing optimization, and routing functions described herein. For example, program instructions may control the operation of an operating system and / or one or more applications. One or more memories may also be configured to store tables such as mobility binding, registration, and association tables. Memory 606 may also hold various software containers and virtualized execution environments and data.

[0110] Network device 600 may also include an application-specific integrated circuit (ASIC) that can be configured to perform routing and / or switching operations. The ASIC can communicate with other components in network device 600 via bus 610 to exchange data and signals and coordinate various types of operations of network device 600, such as routing, switching, and / or data storage operations.

[0111] Figure 7 An example computing system architecture of system 700 is shown, which can be used to handle data manipulation and requests, store and move data items, coordinate data placement actions, and perform other computational operations. In this example, the components of system 700 communicate electrically with each other using a connection 706 such as a bus. System 700 includes a processing unit (CPU or processor) 704 and connection 706, which couples various system components, including memory 720 (e.g., read-only memory (ROM) 718 and random access memory (RAM) 716), to processor 704. System 700 may include caches of high-speed memory that are directly connected to, adjacent to, or integrated into processor 704. System 700 can copy data from memory 720 and / or storage device 708 to cache 702 for fast access by processor 704. In this way, cache can provide performance improvements and avoid latency for processor 704 while waiting for data. These and other modules can control or be configured to control processor 704 to perform various actions. Other memory 720 may also be used. Memory 720 may include various different types of memory with different performance characteristics. Processor 704 may include any general-purpose processor and hardware or software services, such as services 1 710, service 2 712, and service 3 714 stored in storage device 708, which are configured to control processor 704 as well as dedicated processors (where software instructions are incorporated into the actual processor design). Processor 704 may be a completely independent computing system, containing multiple cores or processors, buses, memory controllers, caches, etc. Multi-core processors may be symmetric or asymmetric.

[0112] To enable user interaction with computing system 700, input device 722 can represent any number of input mechanisms, such as a microphone for voice, a touch-sensitive screen for gesture or graphical input, a keyboard, a mouse, motion input, voice input, etc. Output device 724 can also be one or more of a variety of output mechanisms known to those skilled in the art. In some instances, a multimodal system allows the user to provide multiple types of input to communicate with computing system 700. Communication interface 726 typically controls and manages user input and system output. There are no limitations on operation on any particular hardware configuration, therefore the basic features described herein can be readily replaced with improved hardware or firmware arrangements under development.

[0113] Storage device 708 is a non-volatile memory and may be a hard disk or other type of computer-readable medium that can store data accessible by a computer, such as a magnetic tape cassette, flash memory card, solid-state storage device, digital multifunction disk, magnetic tape cassette, random access memory (RAM) 716, read-only memory (ROM) 718, and combinations thereof.

[0114] Storage device 708 may include services 710, 712, and 714 for controlling processor 704. Other hardware or software modules may also be considered. Storage device 708 may be connected to connection 706. In one aspect, a hardware module performing a specific function may include software components stored in a computer-readable medium, as well as necessary hardware components, such as processor 704, connection 706, output device 724, etc., to perform that function.

[0115] For clarity, in some instances, this technology may be presented as comprising individual functional blocks, including devices, device components, steps or routines in a software-embodied method, or a combination of hardware and software.

[0116] In some embodiments, computer-readable storage devices, media, and memories may include cables or wireless signals containing bit streams, etc. However, when referred to, non-transitory computer-readable storage media explicitly exclude media such as energy, carrier signals, electromagnetic waves, and the signals themselves.

[0117] The methods according to the examples above can be implemented using computer-executable instructions stored in or available from a computer-readable medium. Such instructions may include, for example, instructions and data that cause or otherwise configure a general-purpose computer, special-purpose computer, or special-purpose processing device to perform a function or a set of functions. Some of the computer resources used may be accessible via a network. The computer-executable instructions may be, for example, binary, intermediate format instructions, such as assembly language, firmware, or source code. Examples of computer-readable media that may be used to store instructions, information, and / or information created during the methods according to the examples include hard disks or optical disks, flash memory, USB devices equipped with non-volatile memory, network storage devices, etc.

[0118] Devices implementing the methods according to these disclosures may include hardware, firmware, and / or software, and may take any of a variety of form factors. Typical examples of such form factors include laptops, smartphones, small form factor personal computers, personal digital assistants, rack-mount devices, standalone devices, etc. The functionality described herein may also be embodied in peripheral devices or add-on cards. As a further example, such functionality may also be implemented on a circuit board in a different chip or in a different process executed in a single device.

[0119] Instructions, media for conveying such instructions, computing resources for executing such instructions, and other structures for supporting such computing resources are means for providing the functionality described in these disclosures.

[0120] Although various examples and other information have been used to explain aspects within the scope of the appended claims, limitations on the claims should not be implied based on specific features or arrangements in such examples, as those skilled in the art can derive various implementations from these examples. Furthermore, while some subjects may have been described in language specific to structural features and / or method steps, it should be understood that the subjects defined in the appended claims are not necessarily limited to these described features or actions. For example, such functionality may be distributed differently or performed in components other than those identified herein. Rather, the described features and steps are disclosed as examples of components of systems and methods within the scope of the appended claims.

[0121] The claim language of stating "at least one of a group" indicates that one or more members of the group satisfy the claim. For example, the claim language of stating "at least one of A and B" means A, B, or A and B.

Claims

1. A method for managing the placement of data items in a distributed storage system, comprising: Determine the current location of the primary replica of a data item stored on a distributed storage system, wherein the current location of the primary replica of the data item includes the current data storage of multiple data storages on the distributed storage system; Determine an access pattern associated with the primary copy of the data item, the access pattern including one or more starting positions of a set of access requests to the primary copy of the data item received by the distributed storage system, and a corresponding number of access requests received from each of the one or more starting positions. Based on the access pattern associated with the primary copy of the data item, different locations on the distributed storage system for storing the primary copy of the data item are determined, the different locations including different data stores from the plurality of data stores; and The primary copies of the data items are placed in different locations on the distributed storage system. Wherein, the primary copy of the data item represents the authoritative status or version of the data item, and has been copied to the replica data item. Determining the different locations on the distributed storage system used to store the primary replica of the data item includes: Based on the access pattern associated with the primary copy of the data item, identify the starting position of the highest number of access requests associated with the primary copy of the data item from the one or more starting positions; Determine which of the plurality of data stores is closest to the starting position of the highest number of access requests associated with the primary replica of the data item; and The different data storage locations associated with the different locations are determined to be located at the starting position of the highest number of access requests associated with the primary copy of the data item.

2. The method according to claim 1, wherein, The one or more starting locations correspond to one or more client devices, which generate the set of access requests for the primary copy of the data item received by the distributed storage system, and wherein placing the primary copy of the data item at different locations on the distributed storage system includes moving the primary copy of the data item from the current data storage to the different data storage.

3. The method according to claim 1 or 2, wherein, Determining the starting position of the different data storage locations closest to the highest number of access requests associated with the primary copy of the data item includes at least one of the following: Determine that the number of hops between the different data storage locations and the starting location is less than the corresponding number of hops between each of the plurality of data storage locations and each of the one or more remaining locations from the one or more starting locations; and The distance between the different data storage locations and the starting location is determined to be less than the corresponding distance between each of the plurality of data storage locations and each of the one or more remaining locations from the one or more starting locations.

4. The method according to claim 1, wherein, Determining the different locations on the distributed storage system used to store the primary replica of the data item includes: Determine the second current location of the second primary copy of the second data item stored on the distributed storage system, wherein the second current location of the second primary copy of the second data item includes the second current data storage from the plurality of data storages on the distributed storage system; Based on the access mode associated with the primary replica of the data item and the second access mode associated with the second primary replica of the second data item, different locations on the distributed storage system are selected for storing the primary replica of the second data item and the second primary replica of the second data item; and The primary copy of the data item and the second primary copy of the second data item are placed in different locations on the distributed storage system.

5. The method according to claim 4, wherein, The second access mode includes one or more corresponding starting positions of a second set of access requests associated with a second primary copy of the second data item, and a second corresponding number of access requests received from each of the one or more corresponding starting positions, wherein placing the primary copy of the data item and the second primary copy of the second data item in different locations on the distributed storage system includes moving the primary copy of the data item from the current location to the different location, and moving the second primary copy of the second data item from the second current location to the different location.

6. The method according to claim 4 or 5, wherein, The primary copy of the data item includes the first partition of the partitioned dataset, and the second primary copy of the second data item includes the second partition of the partitioned dataset.

7. The method according to claim 1 or 2, further comprising: Determine that the primary copy of the data item and the second data item on the distributed storage system have been accessed together a threshold number of times; And after determining the different locations on the distributed storage system for storing the primary replica of the data item, the second data item is moved from its current corresponding location to the different locations, the current corresponding location including one of the plurality of data storages on the distributed storage system.

8. The method according to claim 1 or 2, further comprising: Determining the primary copy of the data item includes a reference to a specific copy of the second data item stored on the distributed storage system, wherein the specific copy of the second data item includes one of the corresponding primary copy of the second data item or a copy of the corresponding primary copy of the second data item; In response to determining different locations on the distributed storage system for storing a primary copy of the data item and determining that the primary copy of the data item includes the reference to a specific copy of the second data item, different locations on the distributed storage system for storing a specific copy of the second data item are selected; and Move a specific copy of the second data item from a corresponding location on the distributed storage system to a different location on the distributed storage system.

9. The method according to claim 1 or 2, further comprising: Collect information associated with the distributed storage system, including statistics associated with one or more resources, one or more data access restrictions associated with one or more data items on the distributed storage system, one or more events associated with at least one of the distributed storage system and one or more networks associated with the distributed storage system, data access patterns, and network statistics, wherein the one or more resources include at least one of storage nodes, compute nodes, virtual machines, software containers, servers, networks, and networked devices; Based on the information associated with the distributed storage system, determine a data placement action estimated to improve data access performance associated with one or more data items on the distributed storage system or to improve the performance of the distributed storage system. The data placement action includes moving at least one data item from at least one storage location to at least one different storage location, where the at least one storage location and the at least one different storage location include different data stores from the plurality of data stores; and In response to determining the data placement action, the at least one data item is moved from the at least one storage location to the at least one different storage location.

10. A system for managing the placement of data items in a distributed storage system, comprising: One or more processors; as well as At least one computer-readable storage medium storing instructions that, when executed by the one or more processors, cause the one or more processors to: Determine the current location of the primary replica of a data item stored on a distributed storage system, wherein the current location of the primary replica of the data item includes the current data storage of multiple data storages on the distributed storage system; Determine an access pattern associated with the primary copy of the data item, the access pattern including one or more starting positions of a set of access requests to the primary copy of the data item received by the distributed storage system, and a corresponding number of access requests received from each of the one or more starting positions. Based on the access pattern associated with the primary copy of the data item, different locations on the distributed storage system for storing the primary copy of the data item are determined, the different locations including different data stores from the plurality of data stores; and The primary copies of the data items are placed in different locations on the distributed storage system. Wherein, the primary copy of the data item represents the authoritative status or version of the data item, and has been copied to the replica data item. Determining the different locations on the distributed storage system used to store the primary replica of the data item includes: Based on the access pattern associated with the primary copy of the data item, identify the starting position of the highest number of access requests associated with the primary copy of the data item from the one or more starting positions; Determine which of the plurality of data stores is closest to the starting position of the highest number of access requests associated with the primary replica of the data item; and The different data storage locations associated with the different locations are determined to be located at the starting position of the highest number of access requests associated with the primary copy of the data item.

11. The system according to claim 10, wherein, Placing a primary copy of the data item in different locations on the distributed storage system includes moving the primary copy of the data item from the current data storage to the different data storage.

12. The system according to claim 11, wherein, Determining the starting position of the different data storage locations closest to the highest number of access requests associated with the primary copy of the data item includes at least one of the following: Determine that the number of hops between the different data storage locations and the starting location is less than the corresponding number of hops between each of the plurality of data storage locations and each of the one or more remaining locations from the one or more starting locations; and The distance between the different data storage locations and the starting location is determined to be less than the corresponding distance between each of the plurality of data storage locations and each of the one or more remaining locations from the one or more starting locations.

13. The system according to claim 10, wherein, Determining the different locations on the distributed storage system used to store the primary replica of the data item includes: Determine the second current location of the second primary copy of the second data item stored on the distributed storage system, wherein the second current location of the second primary copy of the second data item includes the second current data storage from the plurality of data storages on the distributed storage system; Based on the access mode associated with the primary replica of the data item and the second access mode associated with the second primary replica of the second data item, different locations on the distributed storage system are selected for storing the primary replica of the second data item and the second primary replica of the second data item; and The primary copy of the data item and the second primary copy of the second data item are placed in different locations on the distributed storage system.

14. The system according to claim 13, wherein, The primary copy of the data item includes a first partition of the partitioned dataset, the second primary copy of the second data item includes a second partition of the partitioned dataset, and wherein the second access pattern includes one or more corresponding starting positions of a second set of access requests associated with the second primary copy of the second data item, and a second corresponding number of access requests received from each of the one or more corresponding starting positions.

15. The system according to any one of claims 11 to 14, wherein the at least one computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the system to: The primary copy of the data item and the second data item on the distributed storage system are determined to have been accessed together a threshold number of times; and after determining the different locations on the distributed storage system used to store the primary copy of the data item, the second data item is moved from its current corresponding location to the different location, the current corresponding location including one of the plurality of data storages on the distributed storage system.

16. The system according to any one of claims 11 to 14, wherein the at least one computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the system to: Determining the primary copy of the data item includes a reference to a specific copy of a second data item stored on the distributed storage system, wherein, A specific copy of the second data item includes one of the corresponding primary copy of the second data item or a copy of the corresponding primary copy of the second data item; In response to determining different locations on the distributed storage system for storing a primary copy of the data item and determining that the primary copy of the data item includes the reference to a specific copy of the second data item, different locations on the distributed storage system for storing a specific copy of the second data item are selected; and Move a specific copy of the second data item from a corresponding location on the distributed storage system to a different location on the distributed storage system.

17. The system according to any one of claims 11 to 14, wherein the at least one computer-readable storage medium stores instructions that, when executed by the one or more processors, cause the system to: Collect information associated with the distributed storage system, including statistics associated with one or more resources, one or more data access restrictions associated with one or more data items on the distributed storage system, one or more events associated with at least one of the distributed storage system and one or more networks associated with the distributed storage system, data access patterns, and at least one of network statistics, wherein... The one or more resources include at least one of storage nodes, compute nodes, virtual machines, software containers, servers, networks, and networking devices; Based on information associated with a distributed storage system, a data placement action is determined that is estimated to improve data access performance associated with one or more data items on the distributed storage system or to improve the performance of the distributed storage system. The data placement action includes moving at least one data item from at least one storage location to at least one different storage location, the at least one storage location and the at least one different storage location including different data stores from the plurality of data stores. as well as In response to determining the data placement action, the at least one data item is moved from the at least one storage location to the at least one different storage location.

18. A non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, cause the one or more processors to: Determine the current location of the primary replica of the data item stored on the distributed storage system, where, The current location of the primary copy of the data item includes the current data storage of multiple data storages on the distributed storage system; Determine an access pattern associated with the primary copy of the data item, the access pattern including one or more starting positions of a set of access requests to the primary copy of the data item received by the distributed storage system, and a corresponding number of access requests received from each of the one or more starting positions. Based on the access pattern associated with the primary copy of the data item, different locations on the distributed storage system for storing the primary copy of the data item are determined, the different locations including different data stores from the plurality of data stores; and The primary copies of the data items are placed in different locations on the distributed storage system. Wherein, the primary copy of the data item represents the authoritative status or version of the data item, and has been copied to the replica data item. Determining the different locations on the distributed storage system used to store the primary replica of the data item includes: Based on the access pattern associated with the primary copy of the data item, identify the starting position of the highest number of access requests associated with the primary copy of the data item from the one or more starting positions; Determine which of the plurality of data stores is closest to the starting position of the highest number of access requests associated with the primary replica of the data item; and The different data storage locations associated with the different locations are determined to be located at the starting position of the highest number of access requests associated with the primary copy of the data item.

Images