De-duplication data caching method and system using unreliable edge resources

Abstract

The application relates to a deduplicated data caching method and system using unreliable edge resources, which comprises the following steps: acquiring the total storage capacity of all edge servers; searching for candidate caching files by using a similarity-aware hierarchical clustering method to obtain all file clusters after clustering of the candidate caching files; the candidate caching files comprising deduplicated data blocks without repetition; according to the reliability of all edge servers and the file clusters, a heuristic algorithm is used to select file clusters from the file clusters and cache the file clusters into the edge servers until the size of the cached content reaches the total storage capacity. The trade-off between file availability and space efficiency is considered, so that the cache hit rate is effectively improved under the limited edge cache space.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of edge storage, and relates to a deduplicated data caching method and system using unreliable edge resources. BACKGROUND

[0002] With the explosive growth of digital data, deduplication emerges as the times require and receives more and more attention in edge caching to achieve space efficiency. A typical block-level deduplication process is to split files or data streams into fixed-size or variable-size blocks and then calculate their block fingerprints. Only blocks with unique block fingerprints will be stored, while repeated blocks will be deleted. Li et al. model file similarity as a delta-similarity graph, and then assign files with higher popularity (hotness) to the edge on the basis of deduplication-based space reduction. Luo et al. propose a graph-based method to maximize the deduplication rate under delay constraints. LOFS designs a lightweight three-layer hash mapping method to assign the most similar files to the same edge server to better eliminate redundancy. These designs improve space efficiency by exploring the similarity between files, allowing more popular files to be cached at the edge. However, these designs mainly focus on space efficiency, and the edge server is always considered reliable.

[0003] Expanding storage space is another way to make edge caching hold more files. Due to the diversity of edge resources, many studies suggest expanding edge storage space through various available resources, even some unreliable resources. For example, Pu et al. advocate the use of edge caching in cloud radio access networks to facilitate mobile multimedia services. Liu et al. propose an economically efficient edge storage system using embedded storage nodes. Various idle resources and reserved resources are further emphasized in existing literature to achieve cost-effective space expansion. However, in the process of implementing the present application, the inventors found that the aforementioned conventional methods currently have the technical problem of low cache hit rate. SUMMARY

[0004] To solve the above-mentioned problems in the conventional methods, the present application proposes a deduplicated data caching method using unreliable edge resources and a deduplicated data caching system using unreliable edge resources, which can balance between file availability and space efficiency and improve cache hit rate under limited edge cache space.

[0005] To achieve the above-mentioned purposes, the embodiments of the present application adopt the following technical solutions:

[0006] On the one hand, a deduplicated data caching method using unreliable edge resources is provided, comprising the steps of:

[0007] obtaining the total storage capacity of all edge servers;

[0008] searching the candidate cache files by using a similarity-aware hierarchical clustering method to obtain each file cluster after clustering of all the candidate cache files; the candidate cache files include deduplicated data blocks without duplication;

[0009] According to the reliability of all the edge servers and each file cluster, a heuristic algorithm is used to select a file cluster to be cached in an edge server from each file cluster until the size of the cached content reaches the total storage capacity.

[0010] On the other hand, a deduplicated data caching system using unreliable edge resources is also provided, comprising:

[0011] a capacity obtaining module configured to obtain the total storage capacity of all the edge servers;

[0012] a hierarchical clustering module configured to search the candidate cache files by using a similarity-aware hierarchical clustering method to obtain each file cluster after clustering of all the candidate cache files; the candidate cache files include deduplicated data blocks without duplication;

[0013] a selection caching module configured to select a file cluster to be cached in an edge server from each file cluster according to the reliability of all the edge servers and each file cluster by using a heuristic algorithm until the size of the cached content reaches the total storage capacity.

[0014] One of the above technical solutions has the following advantages and beneficial effects:

[0015] The deduplicated data caching method and system using unreliable edge resources, after obtaining the total storage capacity of all the edge servers, searches the candidate cache files of all the deduplicated data blocks by using a similarity-aware hierarchical clustering method, performs hierarchical clustering to obtain each file cluster after clustering of all the candidate cache files, and then selects a file cluster to be cached in an edge server from each file cluster by using a heuristic algorithm until the size of the cached content reaches the total storage capacity. Since the file caching comprehensively considers the popularity of the file, the similarity of the file, and the reliability of the edge server to select the cache file, the availability of the hotter file is improved, the repeated data is deleted and the redundant space is occupied, the space efficiency is improved, the trade-off between the file availability and the space efficiency is considered, and thus the cache hit rate is effectively improved under the limited edge cache space. BRIEF DESCRIPTION OF DRAWINGS

[0016] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0017] Figure 1 Figure 1 is a schematic diagram of one embodiment of a file collection;

[0018] Figure 2 Figure 2 is a schematic diagram of one embodiment of MEAN and its counterpart method of allocation, where (a) is a deduplication-aware data block allocation, (b) is a reliability-aware data block allocation, and (c) is a data block allocation for the MEAN method;

[0019] Figure 3 Figure 3 is a schematic diagram of one embodiment of a flow of a deduplication data caching method using unreliable edge resources;

[0020] Figure 4 Figure 4 is a schematic diagram of one embodiment of the impact of reliability and number of replicas on hit rate, where (a) is the impact of reliability on hit rate, and (b) is the impact of number of replicas on hit rate;

[0021] Figure 5 Figure 5 is a schematic diagram of one embodiment of a hierarchical clustering algorithm, where (a) is a sorted index schematic, and (b) is a clustering result schematic;

[0022] Figure 6 Figure 6 is a schematic diagram of one embodiment of a flow of hierarchical clustering processing;

[0023] Figure 7 Figure 7 is a schematic diagram of one embodiment of the impact of server reliability on caching performance, where (a) is cache hit rate, and (b) is average retrieval latency;

[0024] Figure 8 Figure 8 is a schematic diagram of one embodiment of the impact of total server caching capacity on caching performance, where (a) is cache hit rate, and (b) is average retrieval latency;

[0025] Figure 9 Figure 9 is a schematic diagram of one embodiment of a module structure of a deduplication data caching system using unreliable edge resources. DETAILED DESCRIPTION

[0026] In order to make the objects, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and should not be used to limit the present application.

[0027] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing specific embodiments only and is not intended to be limiting of the present application.

[0028] It is noted that references herein to "embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment can be included in at least one embodiment of the application. The appearances of the phrase in various places in the specification are not necessarily all referring to the same embodiment nor are they necessarily mutually exclusive of one another.

[0029] It will be understood by those within the art that the embodiments described herein can be combined with other embodiments. As used in the description herein and the appended claims, the term "and / or" means any combination of one or more of the associated listed items. The term "comprising" means "including, but not limited to."

[0030] With the booming development of time-sensitive applications such as augmented reality, Internet of Things, and autonomous driving, caching popular files at the network edge has become one of the ubiquitous data operations. This can reduce the number of data requests to remote content providers, especially during peak hours, thereby alleviating network congestion and shortening service latency. One key metric for such a caching system is the hit rate, which quantifies the percentage of data requests that can be handled at the network edge. Therefore, the data of popular files are stored at the network edge servers.

[0031] Currently, the storage space of edge caching is limited and cannot cope with the explosive growth of data. While this can be addressed by renting more dedicated resources, this is not always a cost-effective option for content providers, especially during peak hours. Therefore, some research suggests expanding the edge storage space by integrating various available edge resources, even though some of these resources can be unreliable. These resources can be idle resources provided by various enterprises and individuals, or resources reserved for other applications that are not fully utilized. This provides a cost-effective and fast way for content providers to expand storage space. By storing more files in the expanded space, the hit rate can be improved to some extent. However, the downside is that many of these resources tend to be unreliable. Some edge servers can erase cached content or disconnect at any time. Therefore, to ensure the availability of files, redundant data needs to be generated.

[0032] In contrast, other research proposes using deduplicated storage technology to cache data. One common approach to deduplication is to split files into multiple fixed / variable-sized chunks, with only one copy of each chunk maintained. Deduplication can eliminate redundancy in cached files, improving space efficiency. Reports indicate that in general file-sharing scenarios, a typical space saving is around 50-60%, while data sets with high duplication can see up to 95% optimization or a 20-fold reduction in storage utilization. Since more files can be cached with limited edge storage resources, the hit rate is improved to some extent.

[0033] In real-world storage systems, the above methods should ideally be combined to expand space during idle periods and utilize deduplicated blocks. However, the inventors noted during their research that these two methods are somewhat contradictory. On one hand, the deduplication strategy dictates that any data block should only be stored once; on the other hand, due to the use of unreliable resources, data should be backed up to ensure file availability. It is not uncommon for storage blocks to become unavailable due to hardware failures, software crashes, or reclamation by the application responsible for them. This results in all files sharing that block becoming incomplete and unusable. Therefore, a key question here is: should unreliable space at the network edge be used to store more deduplicated blocks of files, or should redundant blocks be added to address server failures?

[0034] To address the aforementioned challenges, this application proposes an edge caching system (referred to as MEAN) that utilizes unreliable resources and supports deduplication. To achieve a higher hit rate, MEAN avoids extreme strategies (neither backing up all blocks nor deduplicating blocks across all files) and instead adopts a middle ground (copying some blocks and deduplicating the rest). MEAN comprehensively considers file popularity, file similarity, and server reliability when selecting cached files. MEAN improves the availability of more frequently accessed files and consumes less extra space through redundancy, while other files are deduplicated to improve space efficiency.

[0035] Examples of MEAN and its comparison methods include: Figure 1 and Figure 2 As shown, Figure 1 This is a diagram illustrating a collection of files. Figure 1 Eight files (F1 to F8) were divided into 12 blocks (C1 to C12) based on their popularity (9, 2, 1, 4, 1, 3, 2, 1). The cached resources consisted of two edge servers (ES1 and ES2), with a storage size of 5 blocks. The reliability of edge server 1 (ES1) and edge server 2 (ES2) were 0.7 and 0.6, respectively.

[0036] Deduplication-aware data block allocation scheme, such as Figure 2 As shown in (a), the most frequently used files are cached and deduplicated blocks are evenly distributed across edge servers. This approach can improve cache hit rate. However, it is difficult to guarantee the availability of cached files, and the expected cache hit rate is only around 40.7%. h= 0.7 x 0.6 x (9 + 2 + 4 + 1 + 2) + 0.6 x 3 = 9.36, hit rate = 9.36 / 23 40.7%. The inventors found that many distributed storage systems in reality generally use replication to tolerate faults, inspired by this, one of the improvements taken is to add a copy for each hot file, i.e. the reliability-aware data block allocation scheme as shown in Figure 2 (b). Although, due to the reduction of the number of cached files, the expected cache hit rate (about 49.7%) is only slightly improved: E h = (1 - 0.3 x 0.4) x (9 + 4) = 11.44, hit rate = 11.44 / 23 49.7%. The data block allocation scheme of the MEAN method is as shown in Figure 2 (c), is a relatively good solution, with a maximum cache hit rate of about 66.2%: E h = (1 - 0.3 x 0.4) x 9 + 0.7 x (2 + 3 + 2) + 0.6 x 4 = 15.22, hit rate = 15.22 / 23 66.2%; by eliminating part of the redundancy to release cache space and replicating the hottest file F1 to enhance file availability, therefore, Figure 2 (c) can achieve a good trade-off between space efficiency and file availability. The advantage of MEAN is that it can dynamically select cached files and determine the location and number of copies of each cache block, while considering space efficiency and file availability.

[0037] The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0038] Please refer to Figure 3 In one embodiment, the application provides a deduplicated data caching method using unreliable edge resources, including the following processing steps S12 to S16:

[0039] S12, obtain the total storage capacity of all edge servers.

[0040] It can be understood that in the process of performing the data edge storage task, the total storage capacity of all edge servers available in the current application scenario can be known through manual setting or networked automatic reading, etc. In actual research, although both space efficiency and file availability can effectively improve cache hit rate, the combination of the two brings a tricky challenge. In order to explore the relationship between the two, the reliability of different servers and the cache hit rate of block copies were compared by running tests during the research process of the present application. The data set was downloaded from an existing website, and 10 virtual machines (vms) were set as edge cache servers. In each round of experiment, the virtual machines were randomly turned off according to their reliability. Then, 1000 file retrieval requests were generated on a new vm, and if the required file can be retrieved from these vms, it is considered as a hit; otherwise, it is considered as a miss. The result is based on the average value of 100 rounds of experiments.

[0041] As shown in Figure 4 (a), the hit rate under different confidence (reliability) is shown. The first comparison method is a deduplication cache without copies, denoted as NR, and the unique block is randomly distributed in 10 virtual machines. The second is a 3-copy method, denoted as 3R, and each deduplicated block maintains 3 copies in different vms to achieve fault tolerance. As shown in Figure 4 (a), as the server reliability decreases from 1.0 to 0.8, the NR hit rate rapidly decreases from 100% to 7.56%. In contrast, the 3R method still maintains a high level of hit rate (74.80%) when the reliability decreases to 0.8. Therefore, redundancy can have a positive impact on hit rate to a certain extent.

[0042] However, if the block copies are arbitrarily supplemented, the extra space occupied by redundancy will squeeze out the original cache content, which is not conducive to the improvement of cache hit rate. Therefore, further tests were also conducted to observe the impact of the number of copies. The results are shown in Figure 4 (b), in this group of tests, the total storage space of 10 virtual machines is set to 40% of the size of the data set, and the reliability of each virtual machine is fixed at 0.8. When one copy is added for each data block, the cache hit rate starts to grow. However, under the limited storage space, too much redundancy will waste a lot of storage space. Therefore, when the number of data block copies increases from 2 to 5, the hit rate decreases from 28.07% to 13.42%.

[0043] The test results above show that the unreliability of edge servers has a significant negative impact on the hit rate of deduplication caching, and replication is a double-edged sword. Therefore, when implementing deduplication caching using unreliable storage resources, a trade-off must be struck between deduplication and replication. This is a problem that has been difficult to solve in the past, and its complexity increases exponentially when the heterogeneity of file popularity and server reliability is further considered.

[0044] To address this, the proposed MEAN improves file availability through replication and enhances space efficiency by eliminating unnecessary redundancy through deduplication. It supplements data block copies with existing cached content based on the popularity of different files and data sharing dependencies. Furthermore, it considers block location to further improve file availability under heterogeneous server reliability.

[0045] Problem description:

[0046] There is a set of files F = {f1, f2, ...} with a popularity of H. s = {h1, h2, ...}. The set C = {c1, c2, ...} represents the set of all file chunks in F after deduplication, ensuring no duplicate data is generated. Boolean variable x i,j This indicates an inclusion relationship, where x i,j =1 indicates block c j Included in file f i In the middle, a portion of the files in F will be cached at the edge to facilitate data requests and reduce retrieval latency. This deployment can be carried out during periods of low traffic to reduce traffic pressure on the backbone network. The edge resources consist of a set of storage capacities of M... s The system consists of servers S = {S1, S2, ...}, where M1, M2, ... are the initial values. Let the Boolean variable y be... j,k Represents block c j Is it cached on the edge server S? k We will not consider copying a file multiple times on a single edge server, as this does not improve access performance and instead exacerbates data redundancy. Let size(c j ) represents block c j The size of the edge server S k The storage overhead is the total size of the blocks cached on itself, i.e.

[0047] A cached file is considered a hit only when all relevant blocks of the cached file are available at the edge. This depends on two key preconditions: the first precondition is that all relevant file blocks are cached at the edge. (Using α...) i This indicates whether the condition is met. When α i When = 1, file fi Any one chunk c j of f is cached on at least one edge server, that is f i contains chunks. Therefore, the Boolean variable

[0048]

[0049] where the Boolean (Bool) function returns "1" when the variable is not 0.

[0050] The second prerequisite is that for each edge server that caches a chunk c j of f , every chunk of f i should be available on at least one server. In this example, assume that the server reliabilities are R = {rl, r2,...}. Let P(f i , x, y, R) denote the availability of f i under the server reliabilities R, which also depends on the data sharing dependency between cached files (Boolean value x) and the location of the containing chunks and their replicas (Boolean value y). Generally, the value of P(f i , x, y, R) can be roughly estimated as the product of the reliabilities of the caching servers:

[0051]

[0052] where S(i) denotes the smallest set of servers that can cover all chunks of f i .

[0053] According to the Boolean variables mentioned above for files and chunks, the caching problem can be described as follows:

[0054] 1. When a file is cached (a i = 1), all its chunks c j should have at least one replica on the edge storage:

[0055]

[0056] 2. When a chunk c j is not a chunk of any cached file, it is not necessary to be cached on the edge. Moreover, for any chunk that needs to be cached, there are at most |S| replicas on the edge:

[0057]

[0058] 3. The total size of the blocks stored on each edge server cannot exceed its storage capacity:

[0059]

[0060] 4. State variables are all Boolean:

[0061]

[0062] The optimization goal of MEAN is to maximize the cache hit rate, as follows:

[0063]

[0064] This requires a proper trade-off between spatial efficiency and file availability. Spatial efficiency is described as maximizing the number of cached files, i.e., a i . File availability can be represented by maximizing the reliability of each cached file, i.e., P(f i , x, y, R). In addition, due to the low latency feature of edge access, popular files cached at the edge can respond to more data requests per unit time, i.e., h i value is proportional to the cache hit rate. In summary, the cache problem can be described as a problem model with equation (7) as the cache goal and equations (3) to (6) as the constraints.

[0065] S14, search for candidate cache files using a similarity-aware hierarchical clustering method to obtain each file cluster after clustering of all candidate cache files; the candidate cache files include deduplicated non-repeating data blocks;

[0066] S16, according to the reliability of all edge servers and each file cluster, use a heuristic algorithm to select file clusters from each file cluster to cache into edge servers until the size of the cached content reaches the total storage capacity.

[0067] It can be understood that MEAN selects the most profitable cache scheme (cache new files or several duplicate blocks) at each step until the cache space is filled. An ordering index h x ΔP / Δc is defined, which represents the hit rate gain per unit of cache space. The higher the file index ranking, the more beneficial it is to cache, where h represents the popularity (also known as heat) of the file, ΔP represents the increment of file availability, and Δc represents the additional space overhead.

[0068] There are two main drawbacks to directly using ranking metrics to search for candidate cache files: First, the search process is very time-consuming for a large number of candidate files, especially when calculating the Δc value by comparing the number of blocks contained between candidate files and cached content. Furthermore, since the Δc value is updated after each cache decision as the number of cached blocks increases, it is calculated repeatedly. Second, the Δc value is determined based on the difference from existing cached data. If some files form clusters with high popularity but share few blocks with current cached files, these files are unlikely to be selected for caching on the edge, which could potentially reduce the effectiveness of MEAN.

[0069] by Figure 5 (a) For example, based on the sort index, files f1, f2, and f3 will be selected and cached at the edge in sequence. However, clusters f4, f5, and f6, which share many blocks and have high popularity, are not selected by the algorithm to be stored at the edge, despite having a clear caching advantage. The reason is that when selecting the first file, the sort indexes of these three files are relatively large, thus preventing their selection. Moreover, these three files share fewer blocks with the initially selected cached file, ultimately leading to a suboptimal solution.

[0070] To address the two shortcomings mentioned above, similarity-aware hierarchical clustering (SHC) was adopted. SHC is a hierarchical clustering technique that is an iterative clustering process. In each iteration, when the ranking index of two files (clusters) after clustering is greater than their respective previous values, the most similar pair of files is merged into a new cluster. The iteration ends with the clustered file clusters.

[0071] SHC is a feasible and effective method to accelerate cluster generation and index calculation. Based on this, an effective heuristic algorithm is proposed to improve the hit rate of deduplication caching in unreliable edge environments. This application considers three heterogeneous scenarios to illustrate the processing implementation of MEAN, where the latter scenario is a generalization of the former scenario. The heterogeneous scenarios include: 1) all edge servers are reliable; 2) all edge servers have the same reliability; 3) edge servers have heterogeneous reliability.

[0072] The deduplicated data caching method using unreliable edge resources, after obtaining the total storage capacity of all edge servers, searches all candidate cache files of the data blocks without duplication by using a similarity-aware hierarchical clustering method, performs hierarchical clustering on the candidate cache files to obtain each file cluster after clustering of all candidate cache files, and then uses a heuristic algorithm to select file clusters from each file cluster to cache in the edge servers until the size of the cached content reaches the total storage capacity. Since the popularity of the file, the similarity of the file and the reliability of the edge server are comprehensively considered in the selection of the cached file in the file cache, the availability of the hotter file is improved, the space efficiency is improved by deleting the duplicate data and occupying less additional space, the trade-off between file availability and space efficiency is considered, and therefore the cache hit rate is effectively improved under the limited edge cache space.

[0073] In one embodiment, as shown in Figure 6 regarding the above-mentioned step S14, the following processing steps can be included:

[0074] S141, using a hierarchical clustering method based on the Jaccard index, in each iteration of the iterative clustering process, determining whether the ordering index of the two file clusters after clustering is greater than the respective ordering index value before clustering;

[0075] S142, if yes, merging the two files into a new cluster;

[0076] S143, determining the hotness of the new cluster and recalculating the availability probability of the file according to the position of the block in the new cluster;

[0077] S144, after all the iterative clustering is completed, obtaining each file cluster.

[0078] It can be understood that the hierarchical clustering process depends on a similarity function, which indicates which pair of clusters is merged in each iteration. For this purpose, the commonly used Jaccard index is used as the similarity function. For two clusters A and B, their Jaccard value is defined as The hotness h after clustering is the sum of the two original file (cluster) sets, and c is the size of the set after merging the two original file (cluster) sets. The availability probability P of the file can be directly recalculated according to the position of the block in this cluster. The number of sets generated is much smaller than the number of original files, thereby reducing the computational complexity.

[0079] Taking Figure 5 (b) as an example, the files f5 and f6 produce the largest Jaccard value, i.e. They are merged into the first class Φ1. Then, Φ2 and Φ3 are constructed in turn to obtain the final set of three clusters {Φ1,Ф2,f3}.

[0080] In one embodiment, in relation to step S141 described above, further, in the process of calculating the Jaccard index, the intersection and union of two file clusters are calculated by comparing block fingerprints; the block fingerprints are drawn by Bloom filter.

[0081] Specifically, in calculating the Jaccard value, in order to derive the intersection and union of two clusters, the method adopted is to compare block fingerprints, for example, using MD5 or SHA-1 encoding. In addition, information comparison also consumes a large amount of computing resources. For example, for two clusters with |Ф2| and |Ф3| blocks, determining the number of shared blocks requires 0(|Ф2|×|Ф3|) time complexity. In order to further reduce the computational complexity, Bloom Filter (BF) is used to draw the block fingerprints in each cluster. This simplifies the calculation of Jaccard value from two-to-two fingerprint inspection to member query on cluster sketch. The time complexity of the algorithm can be reduced to 0(|Φk BF ), where k BF represents the number of hash functions used. For any block c i in Ф j , if any bit of the k i hash location in the BF vector of Ф BF ' is 1, its BF determines that this block belongs to Ф i '.

[0082] In one embodiment, when the reliability of all edge servers is reliable, in relation to the process of using heuristic algorithm to select file clusters from each file cluster to cache to edge servers until the size of cached content reaches the total storage capacity in step S16 described above, the specific process can include the following processing process:

[0083] Obtain the Bloom filter sketch of each file cluster;

[0084] Calculate the ranking index value of each file cluster according to the Bloom filter sketch;

[0085] Continuously select the file cluster with the largest ranking index value until the size of the cached content reaches the total storage capacity;

[0086] Randomly and uniformly distribute all cached content blocks to each edge server under the limitation of the total storage capacity.

[0087] Specifically, when all edge servers are reliable, i.e. r1=r2=…=1, there is no need to maintain block copies, because any block is available and the server will not crash. Then the index ranking can be directly simplified as In addition, the allocation policy of the cached content will no longer affect the availability of files, as users can retrieve these cached blocks on any edge server. In this case, the maximum cache hit rate can be achieved by caching the most popular files, thus reducing the extra space overhead. The Bloom filter sketch is derived by drawing a Bloom filter.

[0088] The goal of this part is to select a portion of files to cache at the edge, as shown in equation (7). The first cluster with the largest h / c is selected. Subsequently, the ranking indicator h / Δc is calculated for all file clusters Φ i ∈Φ, where Δc=size(Φ i -Ω∩Φ i ) is derived from the intersection operation of the already cached content Ω and the current cluster Φ i . The file cluster with the largest ranking indicator is continuously selected until the size of the cached content size(Ω) reaches the total storage capacity M=∑M k . It is noted that the cached content Ω and the file cluster Φ should be updated in each round of cluster selection. Finally, the blocks of the cached content Ω will be randomly and uniformly distributed to the edge servers under the storage capacity limit.

[0089] In one embodiment, when the reliabilities of all edge servers are the same, the process of selecting file clusters from the file clusters to cache into the edge servers by using the heuristic algorithm in step S16 described above until the size of the cached content reaches the total storage capacity can specifically include the following processing process:

[0090] For any file cluster with high popularity, a complete copy of the file cluster is directly cached to the current edge server.

[0091] Specifically, when all servers have the same reliability, i.e., r1=r2=…=r, the key measure to improve availability is to cache all blocks of a file on a few or even one server. The reason is that, as the number of servers in S(i) (which can cover all blocks of file f i ) decreases, the file availability can be enhanced, as shown in equation (2). When a file cluster Φ i is cached to the current server , there are three different cases. In particular, considering more cases can explore better solutions, but it requires a large number of iteration times. Therefore, the three cases with the most significant hit rate benefits are mainly considered.

[0092] Case one is: caching a complete copy of the file cluster Φ i . For some popular files, some copies can be maintained to ensure their availability. In this case, a complete copy of the file cluster Φ iCache to current edge server The extra space overhead can be calculated as The availability can be calculated from 1-(1-r) λ to 1-(1-r) λ+1 where λ represents the number of copies of file cluster Φ i in the previous servers.

[0093] In one embodiment, when all edge servers have the same reliability, the process of selecting file clusters from each file cluster to cache to edge servers using heuristic algorithm in step S16 above until the size of the cached content reaches the total storage capacity can further include the following processing procedure:

[0094] Cache the part of file cluster that is not cached to the current edge server.

[0095] Specifically, when all edge servers have the same reliability, case two is to cache the part of file cluster Φ i that is not cached. In deduplication-based caching, the part contained in Φ i may have been cached in the previous edge servers. The number of these related servers is denoted by θ. In order to save space, only the remaining part of Φ i can be cached to the current edge server where Ω and represent the content cached by the previous server and the content cached by the current server, respectively. After caching the remaining part of Φ i , the availability of the cluster can be denoted as r θ+1 .

[0096] In one embodiment, when all edge servers have the same reliability, the process of selecting file clusters from each file cluster to cache to edge servers using heuristic algorithm in step S16 above until the size of the cached content reaches the total storage capacity can further include the following processing procedure:

[0097] Cache the deduplicated part of the file cluster that is scattered in other edge servers to the current edge server.

[0098] Specifically, when all edge servers have the same reliability, case three is to cache the part of file cluster Φ i that is cached in all previous servers and does not exist in the current edge server . After caching the remaining part of Φ i , the deduplicated part that is scattered in the previous edge servers can be cached repeatedly to the current edge server This will increase the availability of file cluster f i from rθ+1 The increase to r, and the additional space overhead, i.e., the increased storage, can be expressed as Δc. i =size(Φ i ∩Ω.

[0099] For each edge server s k The content cached in ∈S. Select the file to cache on the first server, and thereafter, for any remaining servers, compute all candidate file clusters Φ. i The sorting index for three cases ∈Φ. Successively select file clusters with the highest sorting index until cached content. Reach its storage capacity

[0100] In one embodiment, when the reliability of all edge servers is heterogeneous, the process in step S16 above, which uses a heuristic algorithm to select file clusters from each file cluster and cache them on the edge servers until the size of the cached content reaches the total storage capacity, may specifically include the following processing steps:

[0101] After sorting the edge servers in descending order of reliability, each file cluster is cached starting from the edge server with the highest reliability, in descending order of sorting index value.

[0102] It's understandable that when all edge servers have heterogeneous reliability (r1≠r2≠…≠r), files with higher ranking metrics should be cached on servers with higher reliability. Otherwise, the availability of these popular files can only be guaranteed by replicas across multiple unreliable servers, which consumes a significant amount of valuable caching resources. Therefore, heterogeneous servers are sorted in descending order of reliability, and files are cached sequentially on these sorted servers. Three different scenarios within this context are analyzed below.

[0103] Furthermore, the aforementioned process of caching each file cluster sequentially from the highest to the lowest sorting index value, starting with the edge server with the highest reliability, may specifically include the following processing steps:

[0104] For any file cluster with high activity, directly cache a complete copy of the file cluster to the current edge server;

[0105] Cache the uncached portions of the file cluster to the current edge server;

[0106] The deduplicated portions of file clusters that are scattered and cached on other edge servers are duplicated and cached on the current edge server.

[0107] Specifically, when all edge servers have heterogeneous reliability, one scenario is: caching complete file clusters Φ ia copy of Φ The extra space overhead can be calculated as Availability can be improved from where Λ i represents the set of servers that have cached a copy of Φ i

[0108] Case two is that there are uncached parts of the file cluster Φ i In a deduplication-based cache, some of the previous servers (denoted by Θ i ) might have cached some of the data blocks of the file cluster Φ i To save space, only the remaining parts of the file cluster Φ i , i.e. The file availability can be obtained by

[0109] Case three is that the file cluster Φ i is cached at all previous servers and there is no part of it that exists in the current edge servers The size of the needed cache can be obtained from Δc i = size(Φ i ∩ Ω), where the file cluster reliability is improved from to

[0110] The edge servers s k in S should be pre-installed in a descending order of their reliabilities, and Δp i should be the aforementioned reliability increments for this embodiment. It can be noted that MEAN can proactively create copies for hot files, which improves their availability and also helps to avoid server hotspots. In this way, the file retrieval requests can be effectively balanced among different servers, thus avoiding the overload of a single server.

[0111] In an embodiment, in order to facilitate the illustration and understanding of the above-mentioned deduplication data caching method using unreliable edge resources, some experimental examples are provided. It should be noted that the experimental examples provided in this embodiment are not a limitation of the above-mentioned method of the present application, but are examples for assisting in understanding the beneficial effects of the above-mentioned method.

[0112] Experimental setup:

[0113] ​​In the experimental example, the prototype system of MEAN includes a cloud and an edge cluster, simulating file retrieval behavior of edge caching. The cloud is deployed on Elastic Compute Service (ECS) with a configuration of 2.5 GHz 8vcpu, 16 GB RAM, 40 GB SSD. ECS runs Ubuntu Linux 16.04x64. edge consists of 11 virtual machines deployed on a desktop PC with a configuration of 3.50 GHz Intel(R) Core(TM) i9-11900K 8-core CPU, 64 GB RAM, 500 GB SSD. Each virtual machine is allocated 4 GB of RAM and a 30 GB virtual disk drive, running Ubuntu Linux 20.04x64. CPU cores are shared by all virtual machines. In the experimental example, 10 virtual machines serve as edge cache servers, and the remaining one serves as a data requester to retrieve files from these edge servers or the cloud. Existing iPerf and ping tools are used to measure network performance. The average of 10 tests shows that the bandwidth between ECS and local virtual machines is 91.6 Mbps with a delay of 29.05 ms, while the bandwidth between any 2 local virtual machines is 1.27 Gbps with a delay of 0.42 ms.

[0114] Dataset: A real dataset from the Github website is used to evaluate the performance of the mean. The dataset consists of source code in.zip format from 357 popular repositories. These repositories are randomly selected under some popular topics, such as Azure, Amazon Web Services, and Docker, etc. Multiple versions of source code are randomly downloaded from each repository. There are a total of 3099 codes (files) in the dataset of, with the largest file size of 12.6 MB and the smallest of 2.74 KB. These files are chunked using a variable-size chunking method, which declares chunk boundaries based on the content of bytes, widely proven to be more efficient than the fixed-size chunking method. The average chunk size is 4.07 KB, and the deduplication ratio (size after deduplication divided by total size) is 46.99%. The popularity of each file is generated by the widely used Zipf distribution.

[0115] Comparison methods:

[0116] HotDedup: is an implementation of the HotDedup algorithm. The popularity of cached files is maximized with capacity constraints. Then these files are deduplicated in a global sense, and the unique partitions are evenly distributed across servers.

[0117] PopF: Select the most popular cached files. This "popularity-first" strategy is widely adopted by many edge caching systems. All chunks of the selected files are stored on one server to improve file availability. Duplicate chunks are eliminated at the server level to improve space efficiency.

[0118] PopF_3R: Incorporating replica tolerance improves file availability based on the PopF approach. The number of replicas is set to 3, which is the default value for many distributed storage systems.

[0119] Cloud_only: It retrieves all requested files from the cloud without considering edge caches.

[0120] Experimental metrics:

[0121] The experimental results are based on the average of 10 rounds of experiments. The default reliability of the 10 servers is set to [0.8; 0.5; 0.7; 0.7; 0.8; 0.6; 0.5; 0.9; 0.5], and their total storage capacity is set to 20% of the dataset size by default. In each round, 500 file retrieval requests are randomly generated based on the popularity of the files in the dataset to evaluate the performance of different methods. Then, according to the reliability, some servers are randomly shut down, and the files in the request list are requested according to the Poisson distribution; the arrival rate λ is set to 90 by default, indicating the expected number of retrieval requests per minute. The metrics include cache hit rate and average file retrieval delay.

[0122] Experimental results:

[0123] The reliability of all edge servers is set to a consistent value, and this value is changed from 0.5 to 1.0 to evaluate its impact on cache performance. When the reliability is 1.0, MEAN adopts the algorithm corresponding to scenario one. Other reliability settings correspond to the implementation algorithm of scenario two. The total cache capacity is set to the default value, which is 20% of the dataset size. The results are shown in Figure 7 .

[0124] HotDedup is most sensitive to server reliability, and its hit rate is low when the server reliability is less than 0.9, as shown in Figure 7 (a). Due to this sensitivity, the average retrieval delay is only slightly lower than Cloud_only, exceeding 0.7 seconds (in Figure 7 (b)). Only when the servers are completely reliable can the highest cache hit rate be achieved. The reason is that the average allocation of data chunks among servers is not conducive to file availability. Server failures will cause a large number of files to be unavailable, severely affecting the cache hit rate.

[0125] PopF and PopF_3R methods have slightly higher cache hit rates than HotDedup method in unreliable environments. Therefore, when the server reliability is lower than 0.9, their cache performance is better than HotDedup. In particular, when the reliability is 0.5, the hit rate of PopF_3R is slightly higher than PopF. The reason is that the 3-copy strategy can improve the availability of files in unreliable environments. However, as the server reliability improves, the performance of the PopF method is significantly better than the PopF method. The reason is that when the server is more reliable, the performance gain from replication will shrink, but its huge space occupation limits the number of edge cache files. In contrast, MEAN can maintain the best cache performance under most reliability settings because MEAN can perform efficient deduplication while adjusting the number of copies according to different reliability scenarios.

[0126] Different cache capacities are set to evaluate their impact on file request performance. The cache capacity of each edge server is increased from 5% to 30% of the dataset size, and the server reliability remains at the default value. MEAN adopts the algorithm in scenario three.

[0127] As Figure 8 (a) shows the impact of edge cache capacity on cache hit rate. MEAN considers both space efficiency and file availability, and has the highest cache hit rate, followed by PopF and PopF_3R. When the edge cache capacity is small, the PopF method has a higher cache hit rate than the PopF_3R method. However, when the cache capacity exceeds 25%, the PopF_3R method has the opposite effect. This is because when the cache space is large enough, multiple copy schemes can improve the availability of files and cope with more server failures. However, both methods cannot effectively utilize file similarity to improve space efficiency. Therefore, their cache hit rates are lower than the average. The hit rate of HotDedup is relatively low, only about 10%, because any failure of a related server will cause a hit failure. When the cache capacity is 30%, the cache hit rate of MEAN and HotDedup differs by up to 77%.

[0128] As Figure 8(b) The average file retrieval latency is shown. As a baseline method, when all the required files are retrieved from the cloud, the average latency is about 0.9 seconds. Caching files at the edge can significantly reduce the latency of the retrieval process. With a cache capacity of only 5% of the dataset size, MEAN can reduce the retrieval latency by more than 50%, to about 0.43s. This gap gradually widens as the edge cache capacity improves. Specifically, when the cache capacity is 30% of the dataset size, the average retrieval latency of MEAN is about 0.15s. This is a reduction of 83% compared to retrieving files from the cloud, and a reduction of 71% compared to HotDedup. This retrieval latency is about half of the PopF and PopF_3R methods.

[0129] Finally, different arrival rates λ are set to evaluate the performance of edge caching for retrieving 100 files. A larger arrival rate corresponds to a high-peak period of intensive file requests. The arrival rate λ is increased from 60 to 210, and then the average retrieval latency and throughput of different methods are measured.

[0130] Table 1 illustrates the average retrieval latency under different arrival rates. As the value of λ increases, the retrieval latency of all methods shows an upward trend, because file requests are continuously accumulated and compete for limited bandwidth resources. In addition, because the bandwidth of the cloud backbone network is much more scarce than that of the edge network, the latency of the Cloud_only method grows faster than other methods. When the arrival rate reaches 210, the average retrieval latency of the Cloud_only method reaches 9.351s, which is nearly 37 times that of the MEAN method.

[0131] Table 1

[0132]

[0133] It should be understood that, although Figure 3 and Figure 6 the steps in the flowchart are shown in sequential order, such that one step necessarily occurs before another, the steps do not have to be executed in the order shown. Unless explicitly stated, the execution of the steps is not necessarily restricted to the sequential order shown in this document. The steps can be executed in other orders. Figure 3 and Figure 6 At least a part of the steps in the flowchart can include multiple sub-steps or multiple stages, which are not necessarily executed at the same time, but can be executed at different times. The execution order of the sub-steps or stages is not necessarily sequential, but can be round-robin or alternating with other steps or sub-steps or stages of other steps.

[0134] Please refer to Figure 9In an embodiment, a deduplicated data caching system 100 using unreliable edge resources is provided, comprising a capacity obtaining module 12, a hierarchical clustering module 14 and a selection caching module 16. The capacity obtaining module 12 is configured to obtain a total storage capacity of all edge servers. The hierarchical clustering module 14 is configured to search candidate caching files using a similarity-aware hierarchical clustering method to obtain file clusters after clustering of all candidate caching files. The candidate caching files include deduplicated data blocks. The selection caching module 16 is configured to select file clusters from the file clusters using a heuristic algorithm according to the reliability of all edge servers and the file clusters, and cache the file clusters into the edge servers until the size of the cached content reaches the total storage capacity.

[0135] The deduplicated data caching system 100 using unreliable edge resources described above, after obtaining the total storage capacity of all edge servers, searches candidate caching files of all deduplicated data blocks using a similarity-aware hierarchical clustering method, clusters the candidate caching files to obtain file clusters after clustering of all candidate caching files, and then selects file clusters from the file clusters using a heuristic algorithm and caches the file clusters into the edge servers until the size of the cached content reaches the total storage capacity. Since the popularity of the files, the similarity of the files and the reliability of the edge servers are comprehensively considered in the file caching to select the cached files, the availability of the hotter files is improved, the space efficiency is improved by deleting the repeated data and occupying less redundant space, and the trade-off between the file availability and the space efficiency is considered, thereby effectively improving the cache hit rate under the limited edge cache space.

[0136] In an embodiment, the deduplicated data caching system 100 using unreliable edge resources described above can also be used to implement the functions corresponding to the steps or sub-steps added in the embodiments of the deduplicated data caching method using unreliable edge resources described above.

[0137] For specific limitations of the deduplicated data caching system 100 using unreliable edge resources, refer to the corresponding limitations of the deduplicated data caching method using unreliable edge resources described above, which will not be repeated here.

[0138] In an embodiment, a computer device is also provided, comprising a memory and a processor. The memory stores a computer program, and the processor implements the following processing steps when executing the computer program: obtaining a total storage capacity of all edge servers; searching candidate caching files using a similarity-aware hierarchical clustering method to obtain file clusters after clustering of all candidate caching files; the candidate caching files include deduplicated data blocks; selecting file clusters from the file clusters using a heuristic algorithm according to the reliability of all edge servers and the file clusters, and caching the file clusters into the edge servers until the size of the cached content reaches the total storage capacity.

[0139] It can be understood that the above computer device includes other software and hardware components not listed in the specification in addition to the above-mentioned memory and processor, and the specific server model can be determined according to different application scenarios, and the specification will not be listed in detail.

[0140] In one embodiment, the processor executing the computer program can also implement the steps or sub-steps added in each embodiment of the above-mentioned deduplicated data caching method using unreliable edge resources.

[0141] In one embodiment, a computer readable storage medium is also provided, which stores a computer program. The computer program is executed by the processor to implement the following processing steps: obtaining the total storage capacity of all edge servers; searching for candidate cache files using the similarity-aware hierarchical clustering method to obtain each file cluster after clustering of all candidate cache files; the candidate cache files include deduplicated data blocks without duplication; according to the reliability of all edge servers and each file cluster, using a heuristic algorithm to select a file cluster from each file cluster to cache in an edge server until the size of the cached content reaches the total storage capacity.

[0142] In one embodiment, the computer program executed by the processor can also implement the steps or sub-steps added in each embodiment of the above-mentioned deduplicated data caching method using unreliable edge resources.

[0143] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiments can be completed by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. Any reference to memory, storage, database or other medium used in the embodiments provided by the present application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration but not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus dynamic random access memory (Rambus DRAM, RDRAM for short) and interface dynamic random access memory (DRDRAM).

[0144] Any combination of the technical features in the above embodiments can be made, and for the sake of brevity, not all possible combinations are described, however, as long as there is no conflict, any combination of the technical features should be considered within the scope of the present disclosure.

[0145] The above embodiments only express several implementation manners of the present application, and the description is relatively specific and detailed, but it should not be understood as a limitation on the scope of the patent. It should be pointed out that for ordinary skilled persons in the art, without departing from the concept of the present application, a number of modifications and improvements can be made, which all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application should be subject to the appended claims.

Claims

1. A method for deduplicating data caching using unreliable edge resources, characterized in that, Including the following steps: Get the total storage capacity of all edge servers; Candidate cache files are searched using a similarity-aware hierarchical clustering method to obtain file clusters after clustering all candidate cache files; the candidate cache files include deduplicated data blocks. Based on the reliability of all edge servers and each of the file clusters, a heuristic algorithm is used to select file clusters from each of the file clusters and cache them in the edge servers until the size of the cached content reaches the total storage capacity. The step of searching for candidate cache files using similarity-aware hierarchical clustering to obtain file clusters after clustering all candidate cache files includes: A hierarchical clustering method based on the Jaccard index is adopted. In each iteration of the iterative clustering process, it is determined whether the ranking index of two files after clustering is greater than their respective ranking index values ​​before clustering. In the process of calculating the Jaccard index, the intersection and union of the two file clusters are calculated by comparing block fingerprints. The block fingerprints are drawn using a Bloom filter. If so, then merge the two files into a new cluster; Determine the popularity of the new cluster and recalculate the usability probability of the file based on the position of the block in the new cluster; After all iterative clustering is completed, the respective file clusters are obtained.

2. The deduplication data caching method using unreliable edge resources according to claim 1, characterized in that, When all edge servers are reliable, the process of selecting file clusters from each of the aforementioned file clusters and caching them on the edge servers using a heuristic algorithm until the size of the cached content reaches the total storage capacity includes: Obtain Bloom filter sketches for each of the aforementioned file clusters; Calculate the sorting index value for each of the file clusters based on the Bloom filter sketch; Continue selecting file clusters with the highest sort index value until the size of the cached content reaches the total storage capacity. Within the limit of the total storage capacity, all cached content blocks are randomly and evenly distributed to each of the edge servers.

3. The deduplication data caching method using unreliable edge resources according to claim 1, characterized in that, When all edge servers have the same reliability, the process of selecting file clusters from each of the aforementioned file clusters and caching them on the edge servers using a heuristic algorithm until the size of the cached content reaches the total storage capacity includes: For any of the file clusters with high popularity, a complete copy of the file cluster is directly cached on the current edge server.

4. The deduplication data caching method using unreliable edge resources according to claim 3, characterized in that, When all edge servers have the same reliability, the process of selecting file clusters from each of the aforementioned file clusters and caching them on the edge servers using a heuristic algorithm until the size of the cached content reaches the total storage capacity also includes: The uncached portions of the file cluster are cached on the current edge server.

5. The deduplication data caching method using unreliable edge resources according to claim 3, characterized in that, When all edge servers have the same reliability, the process of selecting file clusters from each of the aforementioned file clusters and caching them on the edge servers using a heuristic algorithm until the size of the cached content reaches the total storage capacity also includes: The deduplicated portion of the file cluster that is scattered and cached on other edge servers is then cached again on the current edge server.

6. The deduplication data caching method using unreliable edge resources according to claim 1, characterized in that, When the reliability of all edge servers is heterogeneous, the process of selecting file clusters from each of the aforementioned file clusters and caching them in the edge servers using a heuristic algorithm until the size of the cached content reaches the total storage capacity includes: After sorting the edge servers in descending order of reliability, the file clusters are cached starting from the edge server with the highest reliability, in descending order of sorting index values.

7. The deduplication data caching method using unreliable edge resources according to claim 6, characterized in that, The process of caching each file cluster sequentially from the highest to the lowest sorting index value, starting with the edge server with the highest reliability, includes: For any of the file clusters with high popularity, directly cache a complete copy of the file cluster to the current edge server; Cache the uncached portions of the file cluster to the current edge server; The deduplicated portion of the file cluster that is scattered and cached on other edge servers is then cached again on the current edge server.

8. A deduplication data caching system using unreliable edge resources, characterized in that, include: The capacity acquisition module is used to obtain the total storage capacity of all edge servers; The hierarchical clustering module is used to search for candidate cache files using a similarity-aware hierarchical clustering method, and obtain file clusters after clustering all candidate cache files; the candidate cache files include deduplicated data blocks; The selection caching module is used to select file clusters from each of the file clusters and cache them in the edge servers based on the reliability of all edge servers and each of the file clusters using a heuristic algorithm until the size of the cached content reaches the total storage capacity. The hierarchical clustering module employs a Jaccard-based hierarchical clustering approach. In each iteration of the iterative clustering process, it determines whether the ranking index of two files after clustering is greater than their respective ranking index values ​​before clustering. During the calculation of the Jaccard index, the intersection and union of the two file clusters are calculated by comparing block fingerprints, which are drawn using a Bloom filter. If they are found to be true, the two files are merged into a new cluster. The popularity of the new cluster is determined, and the usability probability of the files is recalculated based on the position of the blocks in the new cluster. After all iterative clustering is completed, each file cluster is obtained.

Images