Replica processing method, system, device and medium under cloud storage distributed system

Abstract

The application relates to the technical field of storage, and discloses a replica processing method, system and device under a cloud storage distributed system and a medium, which are applied to a metadata server and include the following steps: receiving a request for obtaining a replica number and a replica access position vector of a sampling sample set sent by a client; the sampling sample set is composed of a plurality of data blocks randomly extracted by the client; returning the replica number set and the replica access position vector set to the client, so that the client determines a replica creation number and a replica position selection set and sends the replica creation number and the replica position selection set to the metadata server; and according to the load condition of a virtual node in the replica position selection set and the replica creation number, a corresponding virtual node is selected to place a replica. In this way, the replicas are placed according to the regional characteristics, the influence of the replicas caused by the multi-region distribution of files is reduced from the system wide range and dynamics, and the virtual node is introduced, so that the system performance is improved and the system load is balanced.

Description

Technical Field

[0001] This invention relates to the field of storage technology, and in particular to a method, system, device and medium for copy processing in a cloud storage distributed system. Background Technology

[0002] With the rapid development of global information technology and the continuous improvement of informatization, the amount of data is growing explosively, requiring petabyte (PB) level massive storage systems to support it. However, due to the limitations of traditional file systems in terms of capacity and performance, the amount of data that can be stored is greatly restricted. Cloud storage has emerged as a solution to this problem, offering advantages such as strong scalability, high cost-effectiveness, and good fault tolerance. The basic platform for cloud storage is a distributed file system. However, the data center distribution of distributed file systems exhibits a certain degree of wide-area and dynamic nature. The multi-regional distribution of files can easily affect the availability of replicas and reduce system reliability. Furthermore, while distributed systems overcome the bottlenecks of computing, capacity, and bandwidth, they also bring about load balancing issues.

[0003] Therefore, how to effectively process replicas in a cloud storage distributed system, improve system reliability, and balance system load is a technical problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0004] The purpose of this invention is to provide a method, system, device, and medium for replica processing in a cloud storage distributed system, which can place replicas in a targeted manner, ensure replica availability, improve system performance, and balance system load.

[0005] To address the aforementioned technical problems, this invention provides a method for replica processing in a cloud storage distributed system, the method comprising:

[0006] The system receives a request from a client to obtain the number of replicas and the replica access location vectors of a sample set; the sample set is composed of multiple data blocks randomly selected by the client.

[0007] The system returns a set of replica counts and a set of replica access location vectors to the client, so that the client can determine the number of replicas to be created and the set of replica location selections based on the set of replica counts and the set of replica access location vectors, and send them to the metadata server.

[0008] Based on the load status of the virtual nodes in the replica location selection set and the number of replicas created, the corresponding virtual nodes are selected for replica placement.

[0009] In a first aspect, in the replica processing method under the cloud storage distributed system provided by the present invention, determining the replica creation quantity and replica location selection set based on the replica number set and the replica access location vector set includes:

[0010] The set of replica counts is analyzed, and the lower bound of the average replica count of multiple data blocks is calculated and used as the replica creation quantity.

[0011] Based on the set of replica access location vectors, a set of replica access location ranges for multiple data blocks is obtained;

[0012] Perform calculations on the set of replica access locations for each data block to determine the set of replica location selections.

[0013] On the other hand, in the replica processing method of the cloud storage distributed system provided by the present invention, the corresponding virtual nodes are selected for replica placement based on the load status of the virtual nodes in the replica location selection set and the number of replicas created, including:

[0014] Obtain the load status of each virtual node in the set of replica locations;

[0015] The virtual nodes in the replica location selection set are sorted according to the obtained load status;

[0016] Based on the sorting results, starting with the virtual node with the lowest load, select virtual nodes in ascending order of load until the number of virtual nodes selected is the same as the number of replicas created.

[0017] The selected virtual nodes are added to a set of virtual nodes and sent to the client so that the client can create replicas based on the set of virtual nodes.

[0018] On the other hand, the replica processing method under the cloud storage distributed system provided by the present invention further includes:

[0019] After a set time period for data access, obtain the availability of the data object;

[0020] Determine whether the availability of the data object is greater than a set threshold;

[0021] If the availability of the data object is greater than a set threshold, then a replication operation is performed on the data object; if the availability of the data object is not greater than the set threshold, then the replication benefit of the data object is calculated.

[0022] After calculating the replica benefit of the data object, the replicas with a replica benefit greater than 0 are deleted, and the new availability of the data object is obtained again.

[0023] Reassess whether the new availability of the data object is greater than a set threshold;

[0024] If the new availability of the data object is greater than a set threshold, the copy is deleted; if the new availability of the data object is not greater than the set threshold, the copy is migrated.

[0025] On the other hand, in the replication processing method under the above-mentioned cloud storage distributed system provided by the present invention, calculating the replication benefit of the data object includes:

[0026] Calculate the cost of maintaining replica consistency for the data object based on the distance between replicas and the number of replicas;

[0027] The price of a virtual node is determined based on its load.

[0028] Calculate the replication cost of the data object based on the maintenance replication consistency cost of the data object and the virtual node price;

[0029] The replication benefit of the data object is obtained based on the replication cost and replication response amount of the data object.

[0030] On the other hand, in the replica processing method under the cloud storage distributed system provided by the present invention, the cost of maintaining replica consistency of the data object is calculated using a first formula; the first formula is:

[0031]

[0032] Among them, rep_con_cost i The cost of maintaining replica consistency for the data object; ω3 represents the cost of updating replicas per unit distance, used to convert the unit of measurement of the consistency cost into a monetary standard; rep_avdis is the average replica distance; r i vn_dis is the number of replicas of the i-th data object; k is the number of replicas, j is the number of virtual nodes, and S is the set of virtual nodes; k,j The distance between replicas;

[0033] The price of the virtual node is obtained using the second formula; the second formula is:

[0034] vn_prise j =ω4·vn_load j ;

[0035] Among them, vn_prise j The virtual node price; ω4 represents the currency unit for converting the percentage of virtual node load into the virtual node price; vn_load jThis indicates the load status of the virtual nodes;

[0036] The cost of replicating the data object is calculated using a third formula; the third formula is:

[0037] rep_cost i =rep_con_cost i +∑ j∈S vn_prise j ;

[0038] Among them, rep_cost i The cost of replicating the data object;

[0039] The replication benefit of the data object is obtained using the fourth formula; the fourth formula is:

[0040] rep_profit i =ω5·rep_access i -rep_cost i ;

[0041]

[0042] Among them, rep_profit i The revenue generated from replicating the data object is represented by ω5, which indicates the standard price used to convert access counts into monetary units. This refers to the number of replica responses of the data object on each virtual node in the set of virtual nodes.

[0043] On the other hand, in the replica processing method of the cloud storage distributed system provided by the present invention, the replication operation includes:

[0044] Calculate the average distance between replicas based on the current set of virtual nodes distributed across multiple replicas;

[0045] Determine whether the average distance between the replicas is less than the set standard distance between replicas;

[0046] If yes, then nodes whose distance to the replica node is less than the replica standard distance are selected as the node candidate set; if no, then the range of replica access locations is selected as the node candidate set.

[0047] Copy the virtual node corresponding to the lowest price in the node candidate set;

[0048] Correspondingly, the deletion operation for the copy includes:

[0049] Delete the copy with the lowest copy yield;

[0050] Correspondingly, the migration operation for the replica includes:

[0051] Migrate the replica with the lowest replication benefit to the virtual node with the lowest price.

[0052] To address the aforementioned technical problems, the present invention also provides a cloud storage distributed system, comprising: a client and a metadata server;

[0053] The client is used to randomly select multiple created data blocks to form a sampling sample set, and send a request to the metadata server to obtain the number of replicas and the replica access location vector of the sampling sample set;

[0054] The metadata server is used to return a set of replica counts and a set of replica access location vectors to the client;

[0055] The client is further configured to determine the number of replicas to be created and the set of replica locations to be selected based on the set of replica counts and the set of replica access location vectors, and send them to the metadata server.

[0056] The metadata server is also used to select the corresponding virtual node for placing the replica based on the load status of the virtual nodes in the replica location selection set and the number of replicas created.

[0057] To address the aforementioned technical problems, the present invention also provides a replica processing device for a cloud storage distributed system, applied to a metadata server, the device comprising:

[0058] Memory, used to store computer programs;

[0059] A processor is used to implement the steps of the above-described replica processing method in a cloud storage distributed system when executing the computer program.

[0060] To address the aforementioned technical problems, the present invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of the aforementioned copy processing method under a cloud storage distributed system.

[0061] As can be seen from the above technical solution, the replica processing method in a cloud storage distributed system provided by the present invention is applied to a metadata server. The method includes: receiving a request from a client to obtain the number of replicas and replica access location vectors of a sample set; the sample set is composed of multiple data blocks randomly selected by the client; returning the replica count set and replica access location vector set to the client, so that the client can determine the number of replicas to be created and the replica location selection set based on the replica count set and the replica access location vector set, and send them to the metadata server; and selecting the corresponding virtual nodes for replica placement based on the load of the virtual nodes in the replica location selection set and the number of replicas to be created.

[0062] The beneficial effects of this invention are as follows: In the replica processing method for a distributed cloud storage system provided by this invention, after the client sends a request to the metadata server, the cloud data server returns a set of replica counts and a set of replica access location vectors to the client. The client then determines the number of replicas to be created and the set of replica location selections. Finally, the metadata server selects the corresponding virtual nodes for replica placement based on the load of the virtual nodes in the replica location selection set and the number of replicas created. Since the multiple created data blocks have geographical characteristics of historical replica access, this method utilizes these geographical characteristics to place replicas in a targeted manner. This reduces the impact of multi-regional file distribution on replicas from the perspectives of system wide-area coverage and dynamism, ensuring replica availability. Furthermore, the introduction of virtual nodes masks the differences between physical nodes, fully considers node load conditions, improves system performance, and balances system load.

[0063] In addition, the present invention also provides a corresponding cloud storage distributed system, a cloud storage distributed system replica processing device and a computer-readable storage medium for the replica processing method under the cloud storage distributed system, which have the same or corresponding technical features as the aforementioned cloud storage distributed system replica processing method and have the same effect. Attached Figure Description

[0064] To more clearly illustrate the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0065] Figure 1 A flowchart of a replica processing method in a cloud storage distributed system provided in an embodiment of the present invention;

[0066] Figure 2 This is a schematic diagram of the structure of a replica processing device in a cloud storage distributed system provided in an embodiment of the present invention. Detailed Implementation

[0067] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the protection scope of the present invention.

[0068] To enable those skilled in the art to better understand the present invention, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 A flowchart of a replica processing method in a cloud storage distributed system provided in an embodiment of the present invention is shown below. Figure 1 As shown, this method is applied to the metadata server of a cloud storage distributed system, including:

[0069] S101, Receive a request from the client to obtain the number of replicas of the sample set and the replica access location vector; the sample set is composed of multiple data blocks randomly selected by the client.

[0070] In practical applications, the cloud storage distributed system involved in this invention can be built on Ceph (a Linux petabyte-scale distributed file system), which allows for easy scaling to petabyte capacity and offers high performance and reliability to support various workloads. The Ceph system includes clients, a metadata server, and storage servers (object storage clusters). Of course, the cloud storage distributed system of this invention can also be built on other types of file systems, as long as they include clients, a metadata server, and a storage server; these will not be elaborated upon here.

[0071] The cloud storage distributed system of this invention can include one metadata server and multiple storage servers, i.e., it adopts a centralized server model, simplifying design and ensuring system scalability. The metadata server manages metadata, data blocks, and namespaces, while also monitoring the status of the storage servers, implementing task allocation, replication strategies, and load balancing. To avoid single-point-of-failure situations with a single metadata server, the system can also be equipped with multiple auxiliary metadata servers for metadata backup. The storage servers are the carriers for storage and reading / writing, completing the tasks assigned by the metadata server and periodically reporting their own status to the metadata server.

[0072] It should be noted that, due to the broad development prospects of cloud storage, while replication technology brings performance improvements to the distributed file system of cloud storage, it also faces problems such as replica placement and dynamic adjustment. In the lifecycle of a replica, the dynamic replication strategy can be divided into two stages: the initial replica creation stage and the dynamic replica adjustment stage. Steps S101 to S103 focus on the initial replica creation stage. For cloud storage applications with read-heavy access characteristics, this invention proposes a replica placement strategy based on user historical information and load (ULPS) corresponding to steps S101 to S103. This strategy fully considers node load conditions and utilizes the geographical characteristics of user historical replica access to place replicas in a targeted manner, improving system performance and balancing system load.

[0073] When executing step S101, it is considered that the extraction of user replica placement features mainly relies on the user's historical information. In a distributed file system, for a user UserA with historical information, the replicas created by them have a certain number of access records. Therefore, this invention introduces two concepts: replica access location vector cn_P and access location range cn_cov, to extract the access features of the replicas created by UserA. The access location vector cn_P is an indicator vector representing the surface of the accessed replica, cn_P = (country number, region number), that is, the access location vector cn_P contains information of the country number and region number. The access location range cn_cov represents the distribution range of clients accessing the same replica. For a data block i with k replica access location records, its access location range is the smallest location range corresponding to more than q replica access location records; if there is no such range, the access location range is the entire set.

[0074] In implementation, firstly, the client randomly selects n data blocks created by UserA to form a sampling sample set N. Then, the client sends a request to the metadata server to obtain the number of replicas of the sampling sample set N and the access location vector cn_P.

[0075] S102. Return the set of replica counts and replica access location vectors to the client so that the client can determine the number of replicas to be created and the set of replica location selections based on the set of replica counts and replica access location vectors and send them to the metadata server.

[0076] In implementation, the cloud data server returns a set of replica counts and a set of replica access location vectors (cn_P) to the client. Then, based on these sets, the client determines the replica creation quantity (rep_num) and the replica location selection set (F), and sends both (rep_num) and (F) to the metadata server to complete the extraction of user replica placement features.

[0077] S103. Based on the load status and number of replicas created in the virtual node selection set, select the corresponding virtual nodes for replica placement.

[0078] It should be noted that in the cloud storage distributed system, this invention classifies nodes into two categories: physical nodes (Node) and virtual nodes (Vnode). A physical node represents a physical machine in a cloud data center; a virtual node represents a node virtualized from a physical machine. Multiple virtual nodes can correspond to the same physical node. Unlike virtual nodes, where network access and load are independent between two physical nodes, two virtual nodes belonging to the same physical node are identical or mutually influential in these aspects. Since the geographical location of the replica node is closely related to data consistency and availability, this invention considers geographical location as an inherent attribute of both physical and virtual nodes. Furthermore, because the geographical location characteristics of a virtual node depend on the physical node it resides in, this invention sets the geographical location characteristics of a virtual node to be the same as those of the physical node it resides in.

[0079] Given the system's domain and dynamic nature, geographical location is used as a fundamental attribute of nodes, with greater emphasis on the impact of multi-regional file system distribution on replication strategies. Virtual node attributes include: storage capacity, used capacity, disk space load, I / O load, geographical location vector, virtual node number, virtual node distance, and virtual node load.

[0080] In implementation, during step S103, after receiving the replica creation count `rep_num` and the replica location selection set `F`, the metadata server can select corresponding virtual nodes (e.g., selecting the same number of virtual nodes as the replica creation count `rep_num` with low load) for replica placement based on the load of the virtual nodes in the replica location selection set `F` and the replica creation count `rep_num`. The number of selected virtual nodes can vary with the replica creation count `rep_num`.

[0081] In the replica processing method of the cloud storage distributed system provided in this embodiment of the invention, after the client sends a request to the metadata server, the cloud data server returns a set of replica counts and a set of replica access location vectors to the client. The client then determines the number of replicas to be created and the set of replica location selections. Finally, the metadata server selects the corresponding virtual nodes for replica placement based on the load of the virtual nodes in the replica location selection set and the number of replicas created. Since the multiple data blocks that have been created have regional characteristics of historical replica access, this regional characteristic is used to place replicas in a targeted manner. This reduces the impact of replicas caused by the multi-regional distribution of files in terms of system wide-area coverage and dynamism, ensuring replica availability. Furthermore, the introduction of virtual nodes masks the differences between physical nodes, fully considers node load conditions, improves system performance, and balances system load.

[0082] Furthermore, in a specific implementation, in the replica processing method under the cloud storage distributed system provided in the embodiments of the present invention, step S102 determines the number of replicas to be created and the set of replica locations based on the set of replica counts and the set of replica access location vectors. Specifically, this may include: analyzing the set of replica counts, calculating the lower bound of the average number of replicas of multiple data blocks and using it as the number of replicas to be created; obtaining the set of replica access location ranges of multiple data blocks based on the set of replica access location vectors; and performing calculations on the set of replica access location ranges of each data block to determine the set of replica locations.

[0083] In implementation, firstly, the client analyzes the set of replica counts, calculates the lower bound of the average number of replicas for the n data blocks, and uses this as the replica creation quantity `rep_num` (the system default lower bound for the number of replicas is 2, meaning there must be at least 2 replicas). Then, based on the replica access location vector set `cn_P`, the client obtains the replica access location range set `cn_cov` for the n data blocks. Finally, the client performs calculations on each data block's replica access location range set `cn_cov` to determine the replica location selection set `F`.

[0084] Furthermore, in a specific implementation, in the replica processing method under the cloud storage distributed system provided in the embodiments of the present invention, step S103 selects the corresponding virtual nodes for replica placement based on the load status and replica creation number of the virtual nodes in the replica location selection set. Specifically, this may include: obtaining the load status of each virtual node in the replica location selection set; sorting the virtual nodes in the replica location selection set according to the obtained load status; selecting virtual nodes sequentially from the virtual node with the lowest load according to the sorting result, starting from the virtual node with the lowest load, until the number of virtual nodes selected is the same as the replica creation number; adding the selected virtual nodes to the virtual node set and sending it to the client so that the client can create replicas based on the virtual node set.

[0085] In implementation, after obtaining the number of replicas created (rep_num) and the replica location selection set (F), the metadata server selects the virtual nodes with low load (rep_num) from the virtual nodes in the replica location selection set (F) for replica placement, thus obtaining the virtual node set (S).

[0086] A higher virtual node load (vn_load) indicates a busier disk and more used space; a lower virtual node load (vn_load) indicates a less idle disk and less used space. Therefore, this invention considers the virtual node load when placing replicas. On the one hand, this avoids data block write operations being blocked due to disk overload, thus preventing system performance degradation; on the other hand, it ensures a more balanced distribution of replicas, reducing the pressure on subsequent replica adjustment operations and saving system resources.

[0087] It should be noted that the above strategy, compared to the replica placement algorithm in highly fault-tolerant systems (Hadoop Distributed File System, HDFS), incorporates node load considerations. Therefore, it can avoid system performance degradation caused by blocking and reduce subsequent replica adjustments. Furthermore, the strategy is tailored to applications within the read set, aiming to improve user access speed and replica availability. By capturing historical user information, replica placement becomes more targeted.

[0088] Steps S101 to S103 above can solve the problems of replica quantity and placement in the initial replica creation phase under a distributed file system model. However, during data access, the system needs to further adjust the replica distribution based on the current state and external factors such as replica access, which is the dynamic replica adjustment phase discussed in detail below. Specifically, during data access, operations such as creating, deleting, and migrating replicas are performed to improve system performance.

[0089] Based on this, in specific implementation, the replica processing method under the cloud storage distributed system provided in the embodiments of the present invention may further include:

[0090] After a set data access period, obtain the availability of data object i (rep_pa). i ;

[0091] Determine the availability of data object i using `rep_pa`. i Is it greater than the set threshold ε?

[0092] If the availability of data object i is rep_pa i If the availability of data object i is greater than the set threshold ε, then a replication operation is performed on the replica; if the availability of data object i is greater than the set threshold ε, then a replication operation is performed on the replica. i If the value is not greater than the set threshold ε, then calculate the replica profit of the data object, rep_profit. i ;

[0093] In calculating the replica profit (rep_profit) of data object i. i Then, the replica profit (rep_profit) will be... i Delete copies with a value greater than 0 and re-acquire the new availability of data object i;

[0094] Reassess whether the new availability of data object i is greater than the set threshold ε;

[0095] If the new availability of data object i is greater than the set threshold ε, then the copy is deleted; if the new availability of data object i is not greater than the set threshold ε, then the copy is migrated.

[0096] In practice, the above process can be used as a copy adjustment strategy, based on the dynamic copy adjustment stage. The file system environment is analogous to the social environment, and data copies are regarded as valuable commodities circulating in the global market environment. Under the guidance of the "invisible hand" of the market, resource utilization is improved and distribution is optimized to a certain extent.

[0097] In practical applications, this involves performing copy, delete, and migrate operations on replicas, as well as obtaining replica profits (rep_profit). i Once the number of replicas is less than 0, the replica can be reset and committed, and the process of dynamically adjusting the replicas will repeat within the set data access time period.

[0098] Furthermore, in specific implementations, in the replica processing method under the cloud storage distributed system provided in the embodiments of the present invention, the step of calculating the replica benefit of the data object may specifically include:

[0099] Step 1: Calculate the cost of maintaining replica consistency for data objects based on the distance between replicas and the number of replicas.

[0100] It's important to add that high consistency is a goal pursued by all distributed systems, and different application services have different consistency requirements. Since maintaining consistency incurs significant overhead, when treating replicas as commodities, the cost of consistency maintenance must be included as part of their overall cost. This ensures that the system fully considers consistency costs when selecting and adjusting replicas. In a distributed system model, write updates triggered on each replica need to be propagated to other replicas. A wide distribution of replicas increases communication overhead and the risk of inconsistency. The cost of consistency maintenance is closely related to the distance between replicas and the number of replicas. That is, under the same conditions, the sparser the geographical distribution, the higher the availability, but the greater the cost of consistency maintenance, and the lower the data consistency. Therefore, defining a system with r... i The cost of maintaining replica consistency for a data object i with replicas is:

[0101]

[0102] Among them, rep_con_cost i ω3 represents the cost of maintaining replica consistency for data objects; ω3 represents the cost of updating replicas per unit distance, used to convert the unit of measurement for consistency maintenance cost into a monetary standard; rep_avdis is the average replica distance, which is set to a constant here for simplicity, since the replica size is a fixed value; r i vn_dis is the number of replicas of the i-th data object; k is the number of replicas, j is the number of virtual nodes, and S is the set of virtual nodes; k,j The distance between replicas is shown in the formula above. It is clear that both the number of replicas and the density of replicas are factors affecting the consistency cost.

[0103] Step 2: Obtain the virtual node price based on the virtual node's load.

[0104] It should be noted that in distributed file system models, to better reflect the principle of load balancing, the concept of virtual nodes is introduced to mask the performance differences of physical nodes. However, this only addresses the unfairness of physical nodes, not the load imbalance. Therefore, this invention defines the concept of virtual node price, using virtual node load as an indicator to evaluate the cost of replicas, thereby guiding the metadata server to select less loaded virtual nodes for replicas, achieving load balancing and minimizing replica costs. For virtual node j in the cloud storage system, its virtual price is vn_prise. j That is, the virtual node load vn_load of virtual node j. jThe increasing function, specifically the virtual node price vn_prise, can be obtained using the following formula. j :

[0105] vn_prise j =ω4·vn_load j ;

[0106] Where ω4 represents the conversion of the percentage of virtual node load into the currency unit of virtual node price; vn_load j This shows the load status of the virtual nodes. It's clear that the higher the load, the higher the price. Naturally, the system won't choose the most expensive virtual nodes as the location for replicas, thus achieving load balancing.

[0107] Step 3: Calculate the replication cost of the data object based on the maintenance replication consistency cost of the data object and the virtual node price.

[0108] It should be noted that introducing a replication mechanism inevitably incurs associated overhead. The cost of replication is defined as being related to the virtual price of a node and the cost of maintaining replication consistency; the former is used for load balancing, and the latter is used to control the overhead of consistency maintenance. Define the ownership of r... i The cost of replicating a data object i is rep_cost. i Specifically, the replication cost of a data object, rep_cost, can be calculated using the following formula. i :

[0109] rep_cost i =rep_con_cost i +∑ j∈S vn_prise j .

[0110] Step 4: Based on the cost and response time of replicating the data object, obtain the revenue from replicating the data object.

[0111] It should be noted that, in order to meet the need for maximizing the revenue of cloud storage services, the concept of replica revenue is introduced. When a replica is accessed, its value as a commodity is realized. Therefore, this invention uses replica response time to define replica revenue. Let data object i in the cloud storage system have r i If each replica is distributed across a set of virtual nodes S = (s1, s2, s3, ...), then the replica profit of data object i is rep_profit. i for:

[0112] rep_profit i =ω5·rep_access i -rep_cost i ;

[0113]

[0114] Among them, rep_profit i For the revenue generated from replicating data objects, ω5 represents the standard price used to convert access counts into monetary units. This represents the replica response rate of a data object across all virtual nodes in the virtual node set. As the formula shows, the benefit of replication is determined by the replication cost and the response rate each replica can provide. Therefore, increasing replication benefit can be achieved by reducing replication costs and improving system performance to increase replica response rate.

[0115] Furthermore, in specific implementation, in the above-mentioned cloud storage distributed system replication method provided in the embodiments of the present invention, the replication operation may specifically include: calculating the average distance between replicas based on the virtual node set information of the current distribution of multiple replicas; determining whether the average distance between replicas is less than the set standard distance between replicas; if so, then selecting nodes whose distance from the node where the replica is located is less than the standard distance between replicas as a node candidate set; if not, then selecting the replica access location range as a node candidate set; and replicating the replica corresponding to the virtual node with the lowest price in the node candidate set.

[0116] It should be noted that when the availability of data object i in the cloud storage system fails to meet the requirements (i.e., is less than or equal to the set threshold ε), the availability of replicas is improved by increasing the number of replicas. Simultaneously, the replication operation also serves two other objectives: First, to further adjust consistency and availability by adding replicas. If the original replicas are widely distributed, the replica range can be appropriately narrowed to prevent excessive consistency maintenance costs. Second, to expand the system access rate by adding replicas. Therefore, the selection of new nodes should focus on node load to better improve replica access speed. The selection of the node j for placing the new replica depends on the density of the replica distribution. If the replicas are widely distributed (i.e., greater than or equal to the replica standard distance rep_stdis), the replica range needs to be concentrated to control the consistency maintenance cost; if the replicas are densely distributed (i.e., less than the replica standard distance rep_stdis), the replica selection needs to be relatively broad to better improve replica availability. It is worth noting that the replica standard distance rep_stdis can be set according to different applications; therefore, the algorithm's adjustment of consistency and availability will differ for applications with different characteristics.

[0117] In implementation, based on the current r iGiven a set of virtual nodes S = (s1, s2, ...) distributed across replicas, calculate the average replica distance rep_avdis. If the average replica distance rep_avdis is greater than or equal to the standard replica distance rep_stdis, the candidate node set G represents the range of replica distribution locations. If the average replica distance rep_avdis is less than the standard replica distance rep_stdis, the candidate node set G consists of nodes whose distance to the node containing replica k is less than rep_avdis. After obtaining the candidate node set G, calculate the virtual node j = {j∈G|min{vn_prise} j This allows for the replication of copies.

[0118] Correspondingly, deleting a copy can specifically include deleting the copy with the lowest copy yield.

[0119] It should be noted that when the availability of data object i in the cloud storage system is satisfied, the cost of replicas can be reduced and the benefits increased by deleting or migrating replicas. If the availability of data object i can still be satisfied after deleting a replica, then the replica with the minimum replica benefit is selected from the set S = (s1, s2, ...) for deletion.

[0120] Correspondingly, the replication operation can include migrating the replica with the lowest replication benefit to the virtual node with the lowest price.

[0121] In implementation, if the availability of data object i in the cloud storage system is satisfied, but deleting the replica would prevent the availability of data object i from being satisfied, then replica migration is performed. This involves migrating low-revenue replicas to low-cost virtual nodes to reduce replica costs and increase replica benefits. The migration operation first deletes the replica, then copies it using the same algorithm used to add replicas, completing the migration.

[0122] This invention, while ensuring replica availability, introduces the costs of market mechanisms and comprehensively considers the balance between load balancing, consistency, and availability to perform adaptive replication, deletion, and migration operations on replicas, thereby minimizing replica overhead and maximizing replica benefits.

[0123] The above embodiments have described in detail the replication method in a cloud storage distributed system. This invention also provides an embodiment corresponding to a cloud storage distributed system. The system includes: a client and a metadata server;

[0124] The client is used to randomly select multiple created data blocks to form a sample set, and send a request to the metadata server to obtain the number of replicas of the sample set and the replica access location vector.

[0125] The metadata server is used to return a set of replica counts and a set of replica access location vectors to the client.

[0126] The client is also used to determine the number of replicas to be created and the set of replica location selections based on the set of replica counts and the set of replica access location vectors, and then send them to the metadata server.

[0127] The metadata server is also used to select the corresponding virtual nodes for replica placement based on the load of the virtual nodes in the replica location selection set and the number of replicas created.

[0128] In the cloud storage distributed system provided in the embodiments of the present invention, the interaction between the client and the metadata server can be used to place the replicas in a targeted manner based on the regional characteristics. This reduces the impact of the replicas caused by the multi-regional distribution of files in terms of the system's wide range and dynamism, ensuring the availability of the replicas. Furthermore, the introduction of virtual nodes masks the differences between physical nodes, fully considers the node load, improves the system performance, and balances the system load.

[0129] Since the embodiments of the system section correspond to the embodiments of the method section, please refer to the description of the embodiments of the method section for the embodiments of the system section, and they will not be repeated here. Furthermore, it has the same beneficial effects as the replica processing method under the cloud storage distributed system mentioned above.

[0130] Furthermore, in a specific implementation, in the cloud storage distributed system provided in the embodiments of the present invention, the client can specifically be used to analyze the set of replica counts, calculate the lower bound of the average number of replicas of multiple data blocks and use it as the replica creation quantity; obtain the set of replica access location ranges of multiple data blocks based on the set of replica access location vectors; and perform calculations on the set of replica access location ranges of each data block to determine the replica location selection set.

[0131] Furthermore, in a specific implementation, in the cloud storage distributed system provided in the embodiments of the present invention, the metadata server can be used to obtain the load status of each virtual node in the replica location selection set; sort the virtual nodes in the replica location selection set according to the obtained load status; according to the sorting result, starting from the virtual node with the lowest load, select virtual nodes in ascending order of load until the number of selected virtual nodes is the same as the number of replicas to be created; add the selected virtual nodes to the virtual node set and send it to the client so that the client can create replicas according to the virtual node set.

[0132] Furthermore, in a specific implementation, in the cloud storage distributed system provided in the embodiments of the present invention, the metadata server can also be used to obtain the availability of data objects after a set time period of data access; determine whether the availability of data objects is greater than a set threshold; if the availability of data objects is greater than the set threshold, then perform a replication operation on the replicas; if the availability of data objects is not greater than the set threshold, then calculate the replica revenue of the data objects; after calculating the replica revenue of the data objects, delete the replicas with a replica revenue greater than 0, and re-obtain the new availability of the data objects; re-determine whether the new availability of the data objects is greater than the set threshold; if the new availability of the data objects is greater than the set threshold, then perform a deletion operation on the replicas; if the new availability of the data objects is not greater than the set threshold, then perform a migration operation on the replicas.

[0133] Based on the same inventive concept, embodiments of the present invention also provide a replica processing device for a cloud storage distributed system. Figure 2 This is a structural diagram of a replica processing device in a cloud storage distributed system provided in an embodiment of the present invention. This embodiment is based on a hardware perspective, such as... Figure 2 As shown, the replica processing devices in a cloud storage distributed system include:

[0134] Memory 20 is used to store computer programs;

[0135] The processor 21 is used to execute computer programs to implement the steps of the replica processing method under the cloud storage distributed system mentioned in the above embodiments.

[0136] The processor 21 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 21 may be implemented using at least one of the following hardware forms: Digital Signal Processor (DSP), Field-Programmable Gate Array (FPGA), or Programmable Logic Array (PLA). The processor 21 may also include a main processor and a coprocessor. The main processor, also known as the CPU, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 21 may integrate a Graphics Processing Unit (GPU), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 21 may also include an Artificial Intelligence (AI) processor, which is used to handle computational operations related to machine learning.

[0137] The memory 20 may include one or more computer-readable storage media, which may be non-transitory. The memory 20 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 20 is used to store at least the following computer program 201, which, after being loaded and executed by the processor 21, is capable of implementing the relevant steps of the copy processing method under the cloud storage distributed system disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 20 may also include an operating system 202 and data 203, etc., and the storage method may be temporary storage or permanent storage. The operating system 202 may include Windows, Unix, Linux, etc. The data 203 may include, but is not limited to, the data involved in the copy processing method under the cloud storage distributed system mentioned above.

[0138] In some embodiments, the copy processing device in a cloud storage distributed system may further include a display screen 22, an input / output interface 23, a communication interface 24, a power supply 25, and a communication bus 26.

[0139] Those skilled in the art will understand that Figure 2 The structure shown does not constitute a limitation on the replica processing device in a cloud storage distributed system and may include more or fewer components than shown.

[0140] The replica processing device under the cloud storage distributed system provided in this embodiment of the invention includes a memory and a processor. When the processor executes the program stored in the memory, it can implement the following method: replica processing method under the cloud storage distributed system, with the same effect as above.

[0141] Finally, the present invention also provides an embodiment corresponding to a computer-readable storage medium. The computer-readable storage medium stores a computer program, which, when executed by a processor, performs the steps described in the above method embodiments.

[0142] It is understood that if the methods in the above embodiments are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and executes all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0143] The computer-readable storage medium provided by this invention includes the aforementioned copy processing method under a cloud storage distributed system, with the same effect.

[0144] It should also be noted that, in this specification, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0145] The foregoing has provided a detailed description of the replication processing method, system, device, and medium in the cloud storage distributed system provided by this invention. The various embodiments in the specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the systems disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple, and relevant parts can be referred to in the method section. It should be noted that those skilled in the art can make several improvements and modifications to this invention without departing from the principles of this invention, and these improvements and modifications also fall within the protection scope of this invention.

Claims

1. A method for replica processing in a cloud storage distributed system, characterized in that, Applied to a metadata server, the method includes: The system receives a request from a client to obtain the number of replicas and the replica access location vectors of a sample set; the sample set is composed of multiple data blocks randomly selected by the client. The system returns a set of replica counts and a set of replica access location vectors to the client, so that the client can analyze the set of replica counts, calculate the lower bound of the average number of replicas for multiple data blocks, and use it as the replica creation quantity. Based on the set of replica access location vectors, a set of replica access location ranges for multiple data blocks is obtained. Each set of replica access location ranges for each data block is processed to determine the replica location selection set. The replica creation quantity and the replica location selection set are then sent to the metadata server. The system obtains the load information of each virtual node in the replica location selection set; sorts the virtual nodes in the replica location selection set according to the obtained load information; based on the sorting result, starting from the virtual node with the lowest load, selects virtual nodes in ascending order of load until the number of selected virtual nodes is the same as the number of replicas to be created; adds the selected virtual nodes to the virtual node set and sends it to the client so that the client can create replicas based on the virtual node set; The relationship between virtual nodes and physical nodes is as follows: multiple virtual nodes correspond to the same physical node, the geographical location characteristics of a virtual node depend on the physical node where the virtual node is located, and the geographical location characteristics of the virtual node and the physical node where the virtual node is located are set to be the same. After a set data access period, the availability of the data object is obtained; it is determined whether the availability of the data object is greater than a set threshold; if the availability of the data object is greater than the set threshold, a replication operation is performed on the replica; if the availability of the data object is not greater than the set threshold, the replica benefit of the data object is calculated; after calculating the replica benefit of the data object, replicas with a replica benefit not greater than 0 are deleted, and the new availability of the data object is obtained again; it is determined again whether the new availability of the data object is greater than the set threshold; if the new availability of the data object is not greater than the set threshold, a deletion operation is performed on the replica; if the new availability of the data object is greater than the set threshold, a migration operation is performed on the replica.

2. A cloud storage distributed system, characterized in that, include: Client and metadata server; The client is used to randomly select multiple created data blocks to form a sampling sample set, and send a request to the metadata server to obtain the number of replicas and the replica access location vector of the sampling sample set; The metadata server is used to return a set of replica counts and a set of replica access location vectors to the client; The client is also used to analyze the set of replica counts, calculate the lower bound of the average number of replicas of multiple data blocks and use it as the replica creation quantity; obtain the set of replica access location ranges of multiple data blocks based on the set of replica access location vectors; perform operations on the set of replica access location ranges of each data block to determine the replica location selection set; and send the replica creation quantity and the replica location selection set to the metadata server. The metadata server is further configured to: obtain the load status of each virtual node in the replica location selection set; sort the virtual nodes in the replica location selection set according to the obtained load status; select virtual nodes sequentially from the lowest load to the highest load according to the sorting result, until the number of selected virtual nodes is the same as the number of replicas to be created; add the selected virtual nodes to the virtual node set and send it to the client so that the client can create replicas based on the virtual node set; wherein, the relationship between virtual nodes and physical nodes is as follows: multiple virtual nodes correspond to the same physical node, the geographical location characteristics of a virtual node depend on the physical node on which the virtual node is located, and the geographical location characteristics of a virtual node are related to the physical node on which the virtual node is located. The geographical location characteristics of the physical nodes are set to be the same; after a set data access time period, the availability of the data object is obtained; it is determined whether the availability of the data object is greater than a set threshold; if the availability of the data object is greater than the set threshold, a replication operation is performed on the replica; if the availability of the data object is not greater than the set threshold, the replica benefit of the data object is calculated; after calculating the replica benefit of the data object, replicas with a replica benefit not greater than 0 are deleted, and the new availability of the data object is obtained again; it is re-determined whether the new availability of the data object is greater than the set threshold; if the new availability of the data object is not greater than the set threshold, a deletion operation is performed on the replica; if the new availability of the data object is greater than the set threshold, a migration operation is performed on the replica.

3. A replica processing device for a cloud storage distributed system, characterized in that, The device is used in a metadata server and includes: Memory, used to store computer programs; A processor, configured to implement the steps of the replica processing method in a cloud storage distributed system as described in claim 1 when executing the computer program.

4. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the copy processing method in the cloud storage distributed system as described in claim 1.

Images