Edge computing deduplication method and system

Abstract

This invention belongs to the field of edge computing technology, specifically disclosing a method and system for edge computing deduplication. The method treats all servers in the edge computing network as agents, with each agent transmitting information to all neighboring agents. An objective function is established using data deletion loss, deduplication benefit, and load balancing benefit. The data deletion loss represents the impact of a deletion scheme between two neighboring servers on the service quality of users within their coverage area. The deduplication benefit measures the retention of local duplicate data after a server deletes locally cached data. The load balancing benefit represents the cache occupancy between two neighboring servers at the network edge after implementing a deletion allocation scheme. Under hard constraints, the objective function is solved to obtain the deduplication scheme. This technical solution interacts with neighboring information, makes decisions based on local and regional information, solves the problem, and protects privacy.

Description

Technical Field

[0001] This invention belongs to the field of edge computing technology and relates to a method and system for deduplicating duplicate data in edge computing. Background Technology

[0002] In recent years, with the continuous development of the Internet of Things (IoT), intelligent mobile devices and mobile networks have been continuously upgraded, and the amount of data collected by various intelligent device sensors is increasing dramatically. Limited storage resources cannot handle the huge amount of data. In order to improve data utilization and reduce data redundancy, deduplication technology has emerged.

[0003] In traditional cloud computing, data deduplication is widely used. Users send data to the cloud, where it performs unified data deletion. This method is convenient, but it cannot ignore the network load and bandwidth consumption caused by the massive data transmission.

[0004] Edge computing, as a computing model that provides intelligent services at the network edge, allows edge servers to replace the cloud in providing computing and storage resources, significantly reducing network load and service latency, and offering a better user experience. However, compared to traditional cloud computing, edge servers have smaller storage space, and this limited storage space can have a significant impact on applications. But the ability of edge servers to obtain data from neighboring servers greatly increases the coverage of single data points, allowing edge servers to access data without requiring storage. When an edge server can obtain the same data from multiple servers, it is considered that duplicate data exists in the edge network. How to achieve deduplication in an edge storage environment has become a problem that needs to be addressed in the field of edge computing.

[0005] For a given edge computing deduplication scenario, there are n edge servers S = {S1, S2, ..., S...} n These edge servers can communicate with each other. The total storage space of each edge server is not the same; a single edge server S... i It has a size of MS i The total storage space. There are n user sets U = {U1, U2, ..., U...} in the scenario. n}, U i For edge server S i The set of users within the coverage area. Since users communicate with only a single server when uploading or accessing data, each user is only within the coverage area of ​​that single server. The edge network caches a total of m data items: Data = {d1, d2, ..., d...}. m Each edge server caches different data; a single edge server S i The initial cached data set is Sdi Edge servers need to provide services to users. When a user within their service area requests popular data that is not cached, they can request data from a neighboring edge server. Before a deletion operation is performed, all user requests are satisfied, and none exceed the maximum data access latency.

[0006] To address the above scenario, a feasible edge deduplication scheme is needed to delete duplicate data while ensuring that all users can still access all data within a given delay, thereby reducing data redundancy.

[0007] A good caching solution needs to consider three key elements: First, deleting locally cached data on the server will inevitably increase latency for users accessing data; however, this increased latency should be minimized. Second, identical data backups should be minimized in the edge storage system, allowing more space to store other data and improving the utilization rate of individual data sets. Third, the load on edge servers in the edge storage system should be as similar as possible, avoiding the situation where all data is cached on a few servers, leading to excessive caching pressure and resource waste. In summary, this caching solution needs to meet three optimization objectives:

[0008] 1. Minimize the overall increase in user access latency in the edge network.

[0009] 2. The data duplication rate in the edge storage system should be minimized.

[0010] 3. The edge storage system should be load-balanced as much as possible, so that the edge servers retain a balanced amount of data after deletion.

[0011] For the edge computing deduplication scenario described above, current solutions all require collecting information from the entire edge network, modeling and solving based on global information. While such solutions can yield relatively optimal solutions, the centralized approach of collecting global information fails to reflect the advantages of edge computing, such as low network load and privacy protection. Summary of the Invention

[0012] The purpose of this invention is to provide a method and system for deduplication in edge computing, so as to solve the problem that existing solutions require the collection of global information for solving and cannot take advantage of the distributed nature of edge computing.

[0013] To achieve the above objectives, the basic solution of the present invention is: an edge computing deduplication method, comprising the following steps:

[0014] In edge computing, all servers are treated as agents, and each agent transmits information with all neighboring agents.

[0015] An objective function is established using data deletion loss, deduplication benefit, and load balancing benefit. The data deletion loss represents the impact of the deletion scheme of two neighboring servers within a local network on the service quality of users within their coverage area. The deduplication benefit represents the retention of local duplicate data after a server deletes local cached data. The load balancing benefit represents the cache usage between two neighboring servers at the network edge after they execute the deletion allocation scheme.

[0016] Under hard constraints, the objective function is solved to obtain a deduplication scheme.

[0017] The working principle and beneficial effects of this basic solution are as follows: This technical solution utilizes the characteristics of edge networks, enabling each server to make decisions independently as a decision node. Edge servers do not need to collect global information; they can make decisions and solve problems solely through information exchange with their neighbors, based on local and regional information. This reduces the computational pressure on decision nodes and the load on the edge network. Avoiding sending all information to the decision center also maximizes the protection of user data privacy cached on the edge servers.

[0018] Furthermore, the objective function λ * for:

[0019]

[0020] Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD i For the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n Let S be the set of decision variables; set S = {S1, S2, ..., S...} n} represents a set of n servers.

[0021] The process involves removing relevant parameters from the data and establishing the objective function, which is simple to perform.

[0022] Furthermore, data deletion loss LD ij The calculation method is as follows:

[0023] N(S i ) for local server S i The set of neighboring servers, l ij Represents agent Si To intelligent agent S j The number of jumps, when l ij When S = 1, it is called S j It is S i The neighbors;

[0024] For agent S i ,S j Using LD ij This represents the loss due to data deletion between the two. Since the loss due to data deletion is directly related to the number of users, |U is used. i | represents S i Number of users within the coverage area:

[0025]

[0026] in, For the initial stage of the intelligent agent S i Access data d k The minimum delay, Represents agent S i Minimum delay after the current delete operation is performed; LD ij The value is non-positive, meaning that the smaller the increase in access latency caused by the deletion scheme, the lower the LD value. ij The larger the value, the better the deletion scheme; i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n} represents the set of decision variables. Indicates the current agent S i Should we decide to delete cached data? k ∈Data, Data={d1,d2,...,d m} represents the entire data set in the edge storage system, U i For edge server S i The set of users within the coverage area.

[0027] Set data deletion loss LD ij Based on user data access method settings, this is used to measure the increase in access latency for local users after the server deletes local cached data.

[0028] Furthermore, the benefits of deduplication (BD) i The calculation method is as follows:

[0029]

[0030] in, For agent S i Get data d kThe total number of sources, when the more identical data is deleted locally in the edge network, the more BD... i The larger the value, the better the deletion scheme; Data = {d1, d2, ..., d...} m} represents the entire data set in the edge storage system.

[0031] The local deduplication rate setting based on the edge storage system is used to measure the retention of local deduplication data after the server deletes local cached data.

[0032] Furthermore, to obtain load balancing benefits BB ij The method is as follows:

[0033]

[0034] Where, γ i For server S i Cache utilization percentage:

[0035]

[0036] Among them, MS i This represents the total storage space on the edge server;

[0037] BB ij According to the two servers S i ,S j Cache usage, with values ​​between (0,1]. When the cache load is balanced between the two servers, BB... ij =1; When the cache load of the two servers differs significantly, BB ij Approaching 0, i.e., BB ij The larger the value, the better the deletion scheme; Data = {d1, d2, ..., d...} m} represents the entire data set in the edge storage system.

[0038] The percentage of cache used on edge servers is used to measure the cache pressure on neighboring servers.

[0039] Furthermore, hard constraints include local hard constraints, and the method for establishing local hard constraints is as follows:

[0040] Edge servers cannot delete data that they do not cache. In the allocation scheme, the data for which an edge server makes a deletion decision should be guaranteed to be in the cache.

[0041]

[0042] in, Indicates the current agent S i Should we decide to delete cached data? k ∈Data, Data={d1,d2,...,dm} represents the set of all data in the edge storage system; set S = {S1, S2, ..., S...} n} represents a set of n servers; Sd i This represents the initial cached data set of the edge server.

[0043] By setting local hard constraints and storing a set of data sources within a local area, the hard constraints are guaranteed to be met, and a data source is provided to compensate for data deletion losses.

[0044] Furthermore, hard constraints also include local hard constraints. The method for establishing local hard constraints is as follows:

[0045] After the edge server makes a deletion decision, the server can still access all data with maximum data access latency, that is:

[0046]

[0047] in, Represents agent S i Get data d k All sources and access delays, of which l ij Represents agent S i To intelligent agent S j hop count; set A set of elements (S) j ,l ij ) represents the intelligent agent S i From the intelligent agent S j With l ij Delayed access to data d k Data = {d1, d2, ..., d} m} represents the entire data set in the edge storage system; For the intermediate variable set, Represents agent S i For all data in l max All possible access delays within, l max The maximum allowable data access latency for the edge network, expressed in hop count.

[0048] By transforming the global optimization objective into a local optimization objective, each server can solve the problem locally.

[0049] Furthermore, the method for solving the objective function includes the following steps:

[0050] S1, Agent S iThe system broadcasts its local cached data set to all neighboring agents via a CacheMessage and receives CacheResponseMessage messages from neighboring agents, storing them in the access delay set. The CacheMessag message includes a local deletion scheme. i The CacheResponseMessage information includes suggestions for all data that the sender of the CacheMessage has already cached, i.e., the sender's feasible value range;

[0051] S2, based on access delay set Calculate the local constraint cost (LocalBenifit) of the edge server. i ;

[0052] S3, Agent S i Find the solution that makes LocalBenifit true while keeping the neighbor agent's values ​​unchanged. i Find the largest new deletion scheme λ′ i Determine whether the local hard constraints are met; if so, calculate the agent S. i New deletion scheme λ′ i The resulting gain is sent to neighboring agents;

[0053] The Gain message is the local gain value Gain(λ′) resulting from the deletion scheme. i It is composed of λ′, and its calculation formula is: Gain(λ′) i ) = LocalBenifit(λ′ i )-LocalBenifit(λ i ), where λ′ i Assign a value to the new round of deletion scheme, λ i Assign a value to the current value;

[0054] S4, Agent S i Receive the Gain of all neighboring agents and compare it with the agent's own Gain value. If the agent's own Gain value is greater than the Gain values ​​of all neighboring agents, then agent S is considered to have completed its calculation. i The deletion operation can be performed, entering the local hard constraint judgment stage. An Ask message is sent to all neighboring agents. The Ask message includes the sending agent's sequence number, whether the cached data can be deleted, and the deletion scheme λ′. i and the gain Gain(λ′) brought by the new deletion scheme i );

[0055] S5, when agent S i After receiving Ask messages from the set of all local data source agents, and comparing them with the set of local variables... Compare, if If no data from the Ask message is deleted, a Response message agreeing to the deletion is returned to the sending neighboring agent.

[0056] If data from the Ask message is deleted, and If there are other paths to this data, then update. Delete this path and replace it with a subsequent path;

[0057] If data from an Ask message is deleted, and there are no other paths for this data in the data delay table, then compare the Gain values ​​in multiple Ask messages, send True to the agent with the highest Gain value, and send False to the other agents; Response indicates whether the agent can perform the current deletion operation, True means yes, False means no;

[0058] S6, when agent S i After receiving all returned Response messages, if any Response message is False, the assignment remains unchanged; if all messages are True, the new deletion scheme λ′ is selected. i Replace the original solution;

[0059] Determine whether the convergence iteration round is met. If the condition is met, the process ends; otherwise, return to step S1.

[0060] By setting local constraints based on the characteristics of edge computing scenarios and the solution objectives, the global optimization objective is transformed into a local optimization objective, thereby adapting to local solutions in a distributed environment.

[0061] Furthermore, the local constraint cost (LocalBenifit) of the edge server is calculated. i The method is as follows:

[0062]

[0063] Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD i For the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n} represents the set of decision variables.

[0064] Calculate the local constraint cost (LocalBenifit) of the edge server. i This allows for local solution processing.

[0065] The present invention also provides an edge computing deduplication system, including multiple servers and a processing module, wherein the processing module executes the method of the present invention to perform deduplication within the edge computing servers.

[0066] This system enables distributed problem-solving, meeting the needs of edge computing for deduplication. Attached Figure Description

[0067] Figure 1 This is a flowchart illustrating the process of solving the objective function of the edge computing deduplication method of the present invention;

[0068] Figure 2 This is a schematic diagram of an edge network for the edge computing deduplication method of the present invention. Detailed Implementation

[0069] Embodiments of the present invention are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0070] In the description of this invention, it should be understood that the terms "longitudinal", "lateral", "up", "down", "front", "rear", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing this invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0071] In the description of this invention, unless otherwise specified and limited, it should be noted that the terms "installation", "connection" and "linking" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two components. They can be direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0072] This invention discloses a method for edge computing deduplication, which realizes distributed solution of edge computing deduplication problem. The edge server does not need to collect global information, but can make decisions and solve the problem by simply interacting with its neighbors and making decisions based on local and local information.

[0073] Table 1. List of Symbol Meanings

[0074]

[0075] The edge computing deduplication method includes the following steps:

[0076] In edge computing, all servers are treated as agents, and each agent transmits information with all neighboring agents.

[0077] An objective function is established using data deletion loss, deduplication benefit, and load balancing benefit. The data deletion loss represents the impact of the deletion scheme of two neighboring servers within a local network on the service quality of users within their coverage area. The deduplication benefit represents the retention of local duplicate data after a server deletes local cached data. The load balancing benefit represents the cache usage between two neighboring servers at the network edge after they execute the deletion allocation scheme.

[0078] Under hard constraints, the objective function is solved to obtain a deduplication scheme.

[0079] In a preferred embodiment of the present invention, the objective function λ * for:

[0080]

[0081] Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD i For the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n Let S be the set of decision variables; set S = {S1, S2, ..., S...} n} represents a set of n servers.

[0082] In a preferred embodiment of the present invention, the data deletion loss LD ij The calculation method is as follows:

[0083] Based on the edge network, users can access local server S i With neighboring server N(S) iTo address the characteristics of data acquisition and the goal of minimizing the overall increase in access latency for users after a deletion operation in edge deduplication scenarios, a data deletion loss constraint is set between servers to represent the loss between two neighboring servers S within a local network area. i ,S j The impact of the deletion plan on the service quality of users within its coverage area.

[0084] N(S i ) for local server S i The set of neighboring servers, l ij Represents agent S i To intelligent agent S j The number of jumps, when l ij When S = 1, it is called S j It is S i The neighbors;

[0085] For agent S i ,S j Using LD ij This represents the loss due to data deletion between the two. Since the loss due to data deletion is directly related to the number of users, |U is used. i | represents S i Number of users within the coverage area:

[0086]

[0087] in, For the initial stage of the intelligent agent S i Access data d k The minimum delay, Represents agent S i Minimum delay after the current delete operation is performed; LD ij The value is non-positive, meaning that the smaller the increase in access latency caused by the deletion scheme, the lower the LD value. ij The larger the value, the better the deletion scheme; i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n} represents the set of decision variables. Indicates the current agent S i Should we decide to delete cached data? k ∈Data, Data={d1,d2,...,d m} represents the entire data set in the edge storage system, U i For edge server S i The set of users within the coverage area.

[0088] The data deletion loss constraint is set based on the user data access method and is used to measure the increase in access latency for local users after the server deletes local cached data.

[0089] In a preferred embodiment of the present invention, the deduplication benefit BD i The calculation method is as follows:

[0090] Based on the goal of edge storage systems to minimize the number of backups of the same data in the edge network, a deduplication efficiency constraint is set for each server. This constraint measures the retention of locally duplicated data after the server deletes locally cached data. This is achieved using the local data deletion rate.

[0091]

[0092] in, For agent S i Get data d k The total number of sources, when the more identical data is deleted locally in the edge network, the more BD... i The larger the value, the better the deletion scheme. The deduplication efficiency constraint is based on the local deduplication rate setting of the edge storage system, which is used to measure the retention of local duplicate data after the server deletes local cached data.

[0093] In a preferred embodiment of the present invention, load balancing benefits BB are obtained. ij The method is as follows:

[0094] Based on the goal of balancing cache usage as much as possible on edge servers, load balancing benefits are set to represent the efficiency of two adjacent servers S at the network edge. i ,S j (l ij =1) Cache usage between the two servers after the deletion allocation scheme is implemented. Load balancing effectiveness is measured by the Jain smoothing index, which is the ratio of cache utilization between the two servers.

[0095]

[0096] Where, γ i For server S i Cache utilization percentage:

[0097]

[0098] Among them, MS i This represents the total storage space on the edge server;

[0099] BB ij According to the two servers S i ,S jCache usage, with values ​​between (0,1]. When the cache load is balanced between the two servers, BB... ij =1; When the cache load of the two servers differs significantly, BB ij Approaching 0, i.e., BB ij The larger the value, the better the deletion strategy. Load balancing benefit constraints are based on the percentage of used cache on edge servers, used to measure the cache pressure on neighboring servers.

[0100] In a preferred embodiment of the present invention, local constraints are established, transforming the global optimization objective into a local optimization objective, enabling each server to solve the problem locally. Hard constraints include local hard constraints, and the method for establishing local hard constraints is as follows:

[0101] Edge servers cannot delete data that they do not cache. In the allocation scheme, the data for which an edge server makes a deletion decision should be guaranteed to be in the cache.

[0102]

[0103] in, Indicates the current agent S i Should we decide to delete cached data? k ∈Data; Set S = {S1, S2, ..., S...} n} represents a set of n servers; Sd i This represents the initial cached data set of the edge server.

[0104] More preferably, hard constraints also include local hard constraints, and the method for establishing local hard constraints is as follows:

[0105] After the edge server makes a deletion decision, the server can still access all data with maximum data access latency, that is:

[0106]

[0107] in, Represents agent S i Get data d k All sources and access delays, of which l ij Represents agent S i To intelligent agent S j hop count; set A set of elements (S) j ,l ij ) represents the intelligent agent S i From the intelligent agent S j With l ij Delayed access to data d k Data = {d1, d2, ..., d} m} represents the entire data set in the edge storage system; For the intermediate variable set, Represents agent S i For all data in l max All possible access delays within, l max The maximum allowable data access latency for the edge network, expressed in hop count.

[0108] In a preferred embodiment of the present invention, the method for solving the objective function specifically includes five messages: CacheMessage and CacheResponseMessage, which broadcast the local caching scheme during the initialization phase; Gain message, which sends its own gain value during the solution phase; Ask message, which queries neighbors to inquire whether the hard constraints are satisfied, and Response message, which replies to neighbors regarding the satisfaction of the hard constraints, during the hard constraint confirmation phase. Figure 1 As shown, the method for solving the objective function includes the following steps:

[0109] S1, Agent S i The system broadcasts its local cached data set to all neighboring agents via a CacheMessage and receives CacheResponseMessage messages from neighboring agents, storing them in the access delay set. The CacheMessag message includes a local deletion scheme. i The CacheResponseMessage information includes suggestions for all data already cached by the sender of the CacheMessage, i.e., the sender's feasible value range; the feasible value range is when the receiver finds that there is only one path to retrieve certain data. k When this happens, the agent for that path will be informed that the data cannot be deleted, meaning that its feasible value range does not include the data d to be deleted. k .

[0110] Agent S i Receive values ​​from neighboring agents and send the feasible value domain to the neighboring agents; for agent S i The set of decision variables controlled by ∈S Each decision variable in Its range is {-1, 0, 1};

[0111] Where 0 indicates that the edge server decides to retain data d k 1 indicates that the edge server has decided to delete data d. k -1 indicates that the edge server has no cached data d. k Furthermore, a variable with a value of -1 cannot change its own value; set S = {S1, S2, ..., S...} n} represents a set of n servers; λi ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n} represents the set of decision variables. Indicates the current agent S i Should we decide to delete cached data? k ∈Data.

[0112] S2, based on access delay set Calculate the local constraint cost (LocalBenifit) of the edge server. i ;

[0113] S3, Agent S i Find the solution that makes LocalBenifit true while keeping the neighbor agent's values ​​unchanged. i Find the largest new deletion scheme λ i '', determine whether the local hard constraint requirements are met; if so, calculate the agent S. i New deletion scheme λ i The gain brought by ′ is sent to neighboring agents;

[0114] The Gain message is the local gain value Gain(λ′) resulting from the deletion scheme. i It is composed of λ′, and its calculation formula is: Gain(λ′) i ) = LocalBenifit(λ′ i )-LocalBenifit(λ i ), where λ′ i Assign a value to the new round of deletion scheme, λ i Assign a value to the current value.

[0115] S4, Agent S i Receive the Gain of all neighboring agents and compare it with the agent's own Gain value. If the agent's own Gain value is greater than the Gain values ​​of all neighboring agents, then agent S is considered to have completed its calculation. i The deletion operation can be performed, entering the local hard constraint judgment stage. An Ask message is sent to all neighboring agents. The Ask message includes the sending agent's sequence number, whether the cached data can be deleted, and the deletion scheme λ′. i and the gain Gain(λ′) brought by the new deletion scheme i );

[0116] S5, when agent S i After receiving Ask messages from the set of all local data source agents, and comparing them with the set of local variables... Compare, if If no data from the Ask message is deleted, a Response message agreeing to the deletion is returned to the sending neighboring agent.

[0117] If data from the Ask message is deleted, and If there are other paths to this data, then update. Delete this path and replace it with a subsequent path;

[0118] If data from an Ask message is deleted, and there are no other paths for this data in the data delay table, then compare the Gain values ​​in multiple Ask messages, send True to the agent with the highest Gain value, and send False to the other agents; Response indicates whether the agent can perform the current deletion operation, True means yes, False means no;

[0119] S6, when agent S i After receiving all returned Response messages, if any Response message is False, the assignment remains unchanged; if all messages are True, the new deletion scheme λ′ is selected. i Replace the original solution;

[0120] Determine whether the convergence iteration cycle is met (the algorithm iterates for a specified number of cycles and stops running after reaching a fixed number of cycles). If the condition is met, the process ends; otherwise, return to step S1.

[0121] The solution algorithm satisfies the soft constraint maximization while also satisfying the global hard constraint through the design of a message-passing mechanism for the hard constraint satisfaction stage, by introducing an intermediate variable set. This approach enables local storage of a set of data sources within a specific local area, ensuring the satisfaction of hard constraints and providing a data source for calculating the data deletion loss. Addressing the issue that the global hard constraints for deduplication in edge computing cannot be solved locally by the agent, a hard constraint satisfaction phase is designed. This phase uses message passing to determine whether the agent's data deletion scheme satisfies the global hard constraints.

[0122] More preferably, the local constraint cost (LocalBenifit) of the edge server is calculated. i The method is as follows:

[0123]

[0124] Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD iFor the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n} represents the set of decision variables.

[0125] For example, such as Figure 2 As shown, the set of intelligent agents is set as S = {S1, S2, ..., S...} n Let U1 be the set of users covered by agent S1, and let Sd1 = {d1, d2, d3} be the initial cached data set of agent S1; let l max =1, meaning the maximum allowable latency for data access is 1 hop. For local hard constraints, it is necessary to guarantee that any agent can access any data d. k The latency should be less than the maximum access latency. max =1. Taking agent S1 as an example, S1 needs to access all five data in S1 and S2.

[0126] For soft constraints

[0127] Data deletion loss in LD 24 For example, when When agent S2 does not cache d2 and S4 deletes d2, the data deletion loss is calculated by substituting the formula: in, It is possible The query returned the results.

[0128] Taking S2 as an example, when deleting d3, the deduplication benefit is:

[0129] Load balancing benefits with B2B 12 For example, when only At that time, the load balancing benefit is

[0130] When l max When = 1, the solution steps of the algorithm are as follows:

[0131] 1. Agent S1 broadcasts a CacheMessage to S2, initially caching local data set Sd1 = {d1, d2, d3}.

[0132] 2. Upon receiving the CacheResponseMessage message from S2, due to local... If S1 is the only data source for d2, then S1 cannot delete d2 subsequently.

[0133] 3. Treat the cached data set of neighbor S2 as the initial allocation, solve for the local deletion scheme λ′ with the minimum LocalBenifit1 and the corresponding LocalBenifit′1, calculate the Gain(λ′1) value, and send it to neighbor S2.

[0134] 4. After receiving the Gain message sent by S2, determine whether its own Gain(λ′1) is greater than Gain(λ′2). If it is greater, switch its own state to pending deletion; if it is less, switch its own state to non-deletable.

[0135] 5. Broadcast an Ask message to S2, including its own sequence number S1, Gain(λ′1), λ′ value, and its own deletion status.

[0136] Simultaneously, an S2 broadcast Ask message was received, along with the local variable set. Compare, if If no data from the Ask message has been deleted, a Response message agreeing to the deletion is returned to S2; if data from the Ask message has been deleted, and If there are other paths related to this data in the table, update the data delay table, delete this path, and replace it with a subsequent path.

[0137] 6. Upon receiving a Response message from S2, perform the deletion operation only if the value is True, and then send a new assignment to S2.

[0138] 7. Determine if the loop termination condition is met. If the condition is met, the loop ends; otherwise, return to step 2 and continue execution.

[0139] This invention also provides an edge computing deduplication system, including multiple servers and a processing module. The processing module executes the method of this invention to perform edge computing deduplication within the servers. This technical solution provides distributed modeling and solution for edge computing deduplication scenarios, enabling servers to solve their cache allocation schemes locally. This avoids the global message collection process of centralized solutions, effectively reducing network load and ensuring user privacy and security.

[0140] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0141] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

Claims

1. A method for edge computing-based deduplication, characterized in that, Includes the following steps: In edge computing, all servers are treated as agents, and each agent transmits information with all neighboring agents. An objective function is established using data deletion loss, deduplication benefit, and load balancing benefit. The data deletion loss represents the impact of the deletion scheme of two neighboring servers within a local network on the service quality of users within their coverage area. The deduplication benefit represents the retention of local duplicate data after a server deletes local cached data. The load balancing benefit represents the cache usage between two neighboring servers at the network edge after they execute the deletion allocation scheme. Under hard constraints, the objective function is solved to obtain a deduplication scheme; objective function λ * for: Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD i For the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n Let S be the set of decision variables; set S = {S1, S2, ..., S...} n } represents a set of n servers; Data deletion loss LD ij The calculation method is as follows: N(S i ) for local server S i The set of neighboring servers, l ij Represents agent S i To intelligent agent S j The number of jumps, when l ij When S = 1, it is called S j It is S i The neighbors; For agent S i ,S j Using LD ij This represents the loss due to data deletion between the two. Since the loss due to data deletion is directly related to the number of users, |U is used. i | represents S i Number of users within the coverage area: in, For the initial stage of the intelligent agent S i Access data d k The minimum delay, Represents agent S i Minimum delay after the current delete operation is performed; LD ij The value is non-positive, meaning that the smaller the increase in access latency caused by the deletion scheme, the lower the LD value. ij The larger the value, the better the deletion scheme; i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n } represents the set of decision variables. Indicates the current agent S i Should we decide to delete cached data? k ∈Data, Data={d1,d2,...,d m } represents the entire data set in the edge storage system, U i For edge server S i The set of users within the coverage area.

2. The edge computing deduplication method as described in claim 1, characterized in that, Deduplication Benefits (BD) i The calculation method is as follows: in, For agent S i Get data d k The total number of sources, when the more identical data is deleted locally in the edge network, the more BD... i The larger the value, the better the deletion scheme; Data = {d1, d2, ..., d...} m } represents the entire data set in the edge storage system.

3. The edge computing deduplication method as described in claim 1, characterized in that, Obtaining load balancing benefits BB ij The method is as follows: Where, γ i For server S i Cache utilization percentage: Among them, MS i This represents the total storage space on the edge server; BB ij According to the two servers S i ,S j Cache usage, with values ​​between (0,1]. When the cache load is balanced between the two servers, BB... ij =1; the greater the difference in cache load between the two servers, the higher the BB value. ij The closer it is to 0, the more BB is to 0. ij The larger the value, the better the deletion scheme; Data = {d1, d2, ..., d...} m } represents the entire data set in the edge storage system.

4. The edge computing deduplication method as described in claim 1, characterized in that, Hard constraints include local hard constraints. The method for establishing local hard constraints is as follows: Edge servers cannot delete data that they do not cache. In the allocation scheme, the data for which an edge server makes a deletion decision should be guaranteed to be in the cache. in, Indicates the current agent S i Should we decide to delete cached data? k ∈Data, Data={d1,d2,...,d m } represents the set of all data in the edge storage system; set S = {S1, S2, ..., S...} n } represents a set of n servers; Sd i This represents the initial cached data set of the edge server.

5. The edge computing deduplication method as described in claim 1, characterized in that, Hard constraints also include local hard constraints. The method for establishing local hard constraints is as follows: After the edge server makes a deletion decision, the server can still access all data with maximum data access latency, that is: in, Represents agent S i Get data d k All sources and access delays, of which l ij Represents agent S i To intelligent agent S j hop count; set A set of elements (S) j ,l ij ) represents the intelligent agent S i From the intelligent agent S j With l ij Delayed access to data d k Data = {d1, d2, ..., d} m } represents the entire data set in the edge storage system; For the intermediate variable set, Represents agent S i For all data in l max All possible access delays within, l max The maximum allowable data access latency for the edge network, expressed in hop count.

6. The edge computing deduplication method as described in claim 1, characterized in that, The method for solving the objective function includes the following steps: S1, Agent S i The system broadcasts its local cached data set to all neighboring agents via a CacheMessage and receives CacheResponseMessage messages from neighboring agents, storing them in the access delay set. In the CacheMessage, the local deletion scheme is included. i The current assignment, CacheResponseMessage information includes suggestions for all data already cached by the sender of the CacheMessage message, i.e., the sender's feasible value range; S2, based on access delay set Calculate the local constraint cost (LocalBenifit) of the edge server. i ; S3, Agent S i Find the solution that makes LocalBenifit true while keeping the neighbor agent's values ​​unchanged. i Find the largest new deletion scheme λ′ i Determine whether the local hard constraints are met; if so, calculate the agent S. i New deletion scheme λ′ i The resulting gain is sent to neighboring agents; The Gain message is the local gain value Gain(λ′) resulting from the deletion scheme. i It is composed of λ′, and its calculation formula is: Gain(λ′) i ) = LocalBenifit(λ′ i )-LocalBenifit(λ i ), where λ′ i Assign a value to the new round of deletion scheme, λ i Assign a value to the current value; S4, Agent S i Receive the Gain of all neighboring agents and compare it with the agent's own Gain value. If the agent's own Gain value is greater than the Gain values ​​of all neighboring agents, then agent S is considered to have completed its calculation. i The deletion operation can be performed, entering the local hard constraint judgment stage. An Ask message is sent to all neighboring agents. The Ask message includes the sending agent's sequence number, whether the cached data can be deleted, and the deletion scheme λ′. i and the gain Gain(λ′) brought by the new deletion scheme i ); S5, when agent S i After receiving Ask messages from the set of all local data source agents, and comparing them with the set of local variables... Compare, if If no data from the Ask message is deleted, a Response message agreeing to the deletion is returned to the sending neighboring agent. If data from the Ask message is deleted, and If there are other paths to this data, then update. Delete this path and replace it with a subsequent path; If data from an Ask message is deleted, and there are no other paths for this data in the data delay table, then compare the Gain values ​​in multiple Ask messages, send True to the agent with the highest Gain value, and send False to the other agents; Response indicates whether the agent can perform the current deletion operation, True means yes, False means no; S6, when agent S i After receiving all returned Response messages, if any Response message is False, the assignment remains unchanged; if all messages are True, the new deletion scheme λ′ is selected. i Replace the original solution; Determine whether the convergence iteration round is met. If the condition is met, the process ends; otherwise, return to step S1.

7. The edge computing deduplication method as described in claim 6, characterized in that, Calculate the local constraint cost (LocalBenifit) of the edge server. i The method is as follows: Among them, c l ,c d ,c b These are the coefficients for data deletion loss, deduplication benefit, and load balancing benefit, respectively; LD ij This indicates the loss due to data deletion between the two; BD i For the benefits of deduplication; BB ij For load balancing benefits; λ i ∈λ represents the agent S i A set of deduplication schemes, λ = {λ1, λ2, ..., λ} n } represents the set of decision variables.

8. An edge computing deduplication system, characterized in that, It includes multiple servers and a processing module, wherein the processing module performs the method described in any one of claims 1-7 to perform edge computing deduplication within the server.

Images