Data cloud edge collaborative management method and system based on public cloud storage service

Abstract

The application provides a data cloud edge collaborative management method and system based on a public cloud storage service, for each data object, taking the cost generated in the storage life cycle as an optimization target and the average delay of all requests as a constraint, a long-term integer programming problem is constructed; the long-term optimization target and the long-term constraint of the long-term integer programming problem are respectively converted into short-term cost optimization and short-term delay optimization of a single time slot, and the two are combined by weighting; according to the incoming request information in each time slot, the short-term problem is approximately solved to obtain an edge copy placement strategy and a corresponding cloud copy storage category. The application overcomes the difficulty of cost optimization under the condition that the prior knowledge of future requests is unknown and the challenge of low efficiency of integer programming solution, and solves the problem of high cost caused by multiple copies in cloud edge collaborative storage.

Description

Technical Field

[0001] This invention relates to the field of cloud storage and edge storage technology, and in particular to a data cloud-edge collaborative management method and system based on public cloud storage services. Background Technology

[0002] The statements in this section are merely background information related to the present invention and do not necessarily constitute prior art.

[0003] In recent years, edge computing has developed rapidly and has grown into a new computing paradigm. Compared with traditional cloud computing, edge computing refers to deploying computing devices at the network edge close to the user, thus providing users with extremely low latency. This has positive implications for improving the end-user experience. Currently, cloud service providers, in addition to providing traditional cloud computing and cloud storage services, are also beginning to offer edge computing and edge storage related services. For example, Alibaba Cloud provides ENS service, supporting users to purchase computing and storage resources deployed at the edge, and promising performance guarantees of an average latency of less than 5ms and fewer than 4 network request hops. For cloud storage users, edge storage can be used to create data replicas, providing users with an ultimate latency experience.

[0004] Unlike cloud services, servers deployed at the network edge have limited service range, typically only covering requests in specific areas. Requests outside their service range result in more hops and thus higher latency. Therefore, to ensure a low-latency experience for end users distributed across different regions, a common practice is to create a data replica in each edge region. Similar to cloud storage services, edge storage services require data owners to pay for the data hosted on them. While this approach guarantees low latency for users in every edge region, the large number of data replicas significantly increases data storage costs. Furthermore, the access patterns for the same data differ across edge regions; it may be frequently accessed in some areas and less frequently in others. Therefore, the placement strategy for data replicas at the edge can be dynamically adjusted based on the different access patterns in different edge regions, minimizing storage costs while providing the lowest possible latency.

[0005] Once a data replica is created in an edge region, requests for that region are fulfilled by the replica in that region, not by the replica in the cloud. Therefore, the placement strategy of data replicas at the edge affects the access patterns of replicas in the cloud. If every edge region has a data replica, all requests are fulfilled by the edge-side replica, and the cloud replica receives zero access. If no edge region has a data replica, all requests are fulfilled by the cloud replica, and the cloud replica's access volume is the sum of all edge region access volumes. Cloud storage services offer various storage categories with different pricing and latency performance. For example, Amazon S3 offers standard storage, infrequent storage, and archive storage, with storage prices decreasing sequentially and access prices increasing sequentially. Standard and infrequent storage offer millisecond-level latency, while archive storage offers minute-level latency. Data owners can dynamically adjust the cloud data replicas between different storage categories. Therefore, to further reduce the cost of using storage services, data owners need to decide not only on the edge replica placement strategy but also adjust the cloud replica's storage category accordingly.

[0006] To determine the most cost-effective edge placement strategy and corresponding cloud storage category, it is necessary to know the access patterns of data in different edge regions throughout its entire lifecycle. However, data owners cannot know the access patterns of data replicas in different edge regions in advance, which poses a challenge to cost and latency optimization. Without knowing future access patterns, blindly creating replicas at the edge and adjusting storage categories in the cloud will not only fail to save costs but may even lead to significant cost waste.

[0007] Existing technologies typically employ greedy or prediction-based approaches, but these technologies can only achieve cost savings in certain scenarios. When access patterns fluctuate drastically or are difficult to predict accurately, they may result in exorbitant costs that are unacceptable to data owners. Summary of the Invention

[0008] To address the shortcomings of existing technologies, this invention provides a data cloud-edge collaborative management method and system based on public cloud storage services. This method eliminates the need to know the access patterns of data replicas in the cloud and at the edge throughout their entire lifecycle. It dynamically adjusts the placement strategy of data replicas at the edge and correspondingly adjusts the storage category of replicas in the cloud. This satisfies end-users' latency requirements while reducing the cost for data owners to use cloud and edge storage services.

[0009] To achieve the above objectives, the present invention adopts the following technical solution: The first aspect of this invention provides a data cloud-edge collaborative management method based on public cloud storage services.

[0010] A data cloud-edge collaborative management method based on public cloud storage services includes the following processes: For a data object, with whether a replica is created in the edge region and the storage type of the replica in the cloud server as decision variables, the portion of the average latency of the current time slot that exceeds the upper limit of the latency acceptable to the end user is calculated, and the cumulative latency exceeding the upper limit of the latency acceptable to the end user up to the current time slot is used as the weight to obtain the first variable; Calculate the cost generated in the current time slot, using user-defined parameters as weights to obtain a second variable, and use the minimum weighted sum of the first and second variables as the objective function; Based on the number of requests in each edge region of the current time slot, the objective function is solved to obtain the placement strategy of this data object in the edge region and its storage category on the cloud server for the current time slot.

[0011] As a further limitation of the first aspect of the present invention, the objective function of each time slot is optimized and solved sequentially throughout the entire life cycle of the data object to obtain the placement strategy of the data object in the edge region and the storage category in the cloud server for each time slot.

[0012] As a further limitation of the first aspect of the present invention, each time slot of the data object is a sub-time period in the entire life cycle. In the entire life cycle, the goal is to minimize the cost generated by using cloud edge storage in the entire life cycle, and the upper limit of the average latency of all end-user requests is constrained. Long-term integer planning is performed across the entire lifecycle, using whether to create replicas in edge regions and the storage category of replicas in the cloud as decision variables. The long-term optimization problem spanning the entire lifecycle is transformed into a short-term optimization problem within each time slot by using the optimization objective and constraints of a time slot under a long-term integer programming over the entire lifecycle. The objective function of each time slot is a weighted sum of the cost and unmet delays of each time slot.

[0013] As a further limitation of the first aspect of the invention, whether or not to create a copy in the edge region is used as a first decision variable, expressed as follows: It is a variable where each item takes the value 0 or 1. dimensional vector, This indicates that no data copy was created for the corresponding edge region. This indicates that a data copy exists in the corresponding edge region.

[0014] As a further limitation of the first aspect of the present invention, the storage category of the replica in the cloud server is a second decision variable, denoted as: 0 represents standard storage, 1 represents low-frequency storage, and 2 represents archive storage.

[0015] As a further limitation of the first aspect of the present invention, the cumulative delay value exceeding the upper limit of acceptable delay for the end user in the current time slot is:

[0016] in, The cumulative latency value exceeding the upper limit of acceptable latency for end users up to the previous time slot. express The time slot decision variable is and The average latency of all requests in the current time slot; This represents the upper bound of the average latency acceptable to end users; For one Dimensional variables, each representing whether a copy is created for a given edge region. This refers to the storage category for data replicas in the cloud.

[0017] As a further limitation of the first aspect of the present invention, the objective function is solved based on the number of requests for each edge region in the current time slot, including: Calculate the objective value of the short-term optimization problem for the current time slot under the current edge replica placement strategy and cloud storage type. ; Randomly select an edge region, modify its placement strategy, and obtain a new edge copy placement strategy. ,calculate The optimal value of the short-term optimization problem in the current time slot And record the storage category of the cloud replica corresponding to the obtained optimal value. ; Based on the target values ​​before and after the edge region placement strategy modification, calculate the value between 0 and 1: ,in This indicates user-defined parameters; by The probability of accepting this change in the edge copy placement strategy is... , ;by The probability of rejecting this change in the edge copy placement strategy is high. , ; If the target value increases by less than 0.1% after repeating the above steps a set number of times, stop the iteration. According to the obtained and Adjust the storage type of the replicas in the edge region and the replicas in the cloud, and traverse... For each term, if it is 0 and If the corresponding item is 1, then the copy in the corresponding edge region will be deleted; if it is 1 and If the corresponding item is 0, then select the nearest edge region with the lowest latency that contains a replica to download the data, and create a new replica in the corresponding edge region.

[0018] A second aspect of the present invention provides a data cloud-edge collaborative management system based on public cloud storage services.

[0019] A data cloud-edge collaborative management system based on public cloud storage services includes: The module includes a price crawler module, a request collection module, a communication module, and a scheduling optimization module. The price crawler module is configured to retrieve price data for edge cloud services and cloud services for each time slot. The request collection module is configured to: obtain the number of requests for each replica in the time slot cloud service and the replicas in its edge region; The communication module is configured to run in all edge storage instances and transmit information between different edge regions. The information transmitted includes the number of times the edge storage requests a certain data object, whether there are data replicas in the edge region, and the edge replica placement strategy. The scheduling optimization module is configured to execute the steps of the data cloud-edge collaborative management method based on public cloud storage services as described in the first aspect of this invention.

[0020] A third aspect of this invention provides a data cloud-edge collaborative management system based on public cloud storage services.

[0021] A data cloud-edge collaborative management system based on public cloud storage services includes: The time slot target calculation unit is configured as follows: For a data object, with whether a replica is created in the edge region and the storage type of the replica in the cloud server as decision variables, the portion of the average latency of the current time slot that exceeds the upper limit of the latency acceptable to the end user is calculated, and the cumulative latency exceeding the upper limit of the latency acceptable to the end user up to the current time slot is used as the weight to obtain the first variable; Calculate the cost generated in the current time slot, using user-defined parameters as weights to obtain a second variable, and use the minimum weighted sum of the first and second variables as the objective function; The objective function solving unit is configured as follows: Based on the number of requests in each edge region of the current time slot, the objective function is solved to obtain the placement strategy of this data object in the edge region and its storage category on the cloud server for the current time slot.

[0022] A fourth aspect of the present invention provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the steps of the data cloud-edge collaborative management method based on public cloud storage services as described in the first aspect of the present invention.

[0023] The fifth aspect of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps in the data cloud-edge collaborative management method based on public cloud storage services as described in the first aspect of the present invention.

[0024] Compared with the prior art, the beneficial effects of the present invention are: 1. This invention innovatively proposes a data cloud-edge collaborative management method and system based on public cloud storage services. It does not require prior knowledge of the access patterns of data replicas in the cloud and edge throughout their entire lifecycle. It dynamically adjusts the placement strategy of data replicas at the edge and correspondingly adjusts the storage category of replicas in the cloud. This satisfies the latency requirements of end users while reducing the cost for data owners to use cloud storage services and edge storage services.

[0025] 2. This invention innovatively proposes a data cloud-edge collaborative management method and system based on public cloud storage services. It transforms a long-term optimization problem that requires future access volume into a short-term optimization problem that does not require future access volume. It can quickly solve large-scale edge storage areas and make cost-effective edge replica placement strategies and cloud replica storage category migration strategies based on the real-time access volume of different edge areas. It can ensure the latency requirements of end users while minimizing the cost for data owners, thereby achieving cost-effective data cloud-edge collaborative management. Attached Figure Description

[0026] The accompanying drawings, which form part of this invention, are used to provide a further understanding of the invention. The illustrative embodiments of the invention and their descriptions are used to explain the invention and do not constitute an improper limitation of the invention.

[0027] Figure 1 This is a schematic diagram of the data cloud-edge collaborative management method based on public cloud storage services provided in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the architecture of the data cloud-edge collaborative management system based on public cloud storage services provided in Embodiment 2 of the present invention. Detailed Implementation

[0028] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0029] It should be noted that the following detailed description is illustrative and intended to provide further explanation of the invention. Unless otherwise specified, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains.

[0030] It should be noted that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of exemplary embodiments according to the invention. As used herein, the singular form is intended to include the plural form as well, unless the context clearly indicates otherwise. Furthermore, it should be understood that when the terms "comprising" and / or "including" are used in this specification, they indicate the presence of features, steps, operations, devices, components, and / or combinations thereof.

[0031] Where there is no conflict, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0032] Example 1: like Figure 1 As shown, Embodiment 1 of the present invention provides a data cloud-edge collaborative management method based on public cloud storage services. Taking Amazon S3 service and Alibaba Cloud ENS service as examples, data stored in S3 is accessible to all regions, while data stored in ENS only serves the edge region where it is located. For all replicas of each data object, in order to optimize the cost incurred throughout its entire lifecycle while ensuring the request latency of end users, the following steps are performed: Step 1: Using the cost incurred by all replicas of this data object over its entire lifecycle as the optimization objective and the average latency of all requests from end users as the constraint, construct an integer programming problem, formalized as follows: (1) in, For one The variables are dimensional, each representing whether a copy is created for a given edge region. Each item takes a value of 0 or 1, indicating that no copy is created and a copy is created, respectively. This indicates the storage category of data replicas in the cloud: 0 represents Standard storage, 1 represents Standard-IA storage, and 2 represents Glacier Instant Retrieval. express The time slot decision variable is and The costs incurred at that time; express The time slot decision variable is and The average latency of all requests in the current time slot; This represents the upper bound of the average latency acceptable to end users; for It consists of two parts: the cost generated by S3. Costs generated by ENS .

[0033] S3 cost It consists of 5 parts, namely storage cost Request cost Data retrieval costs Bandwidth cost Migration costs .

[0034] The storage cost is calculated as follows: (2) For storage category Storage unit price at that time This refers to the size of the data replica in the cloud. The request cost is calculated as follows: (3) For storage category The unit price requested at that time, for The number of requests to replicas in the time-slotted cloud; The cost of data retrieval is calculated as follows: (4) For storage category The unit price for data retrieval at that time for The number of requests to replicas in the time-slotted cloud, which is also the number of times the data is retrieved; Bandwidth cost is calculated as follows: (5) This is the unit price for bandwidth, expressed per GB; migration costs only arise when the storage category changes, and are calculated as follows: (6) Indicates from storage category The unit price for migration operations when migrating out.

[0035] ENS cost It consists of two parts: the storage cost of all ENS storage instances. The bandwidth cost for all ENS storage instances .

[0036] For an edge ENS instance, the storage cost is calculated as follows: (7) This is the unit price for ENS storage, expressed as per GB per month. Bandwidth cost is calculated as follows: (8) This is the unit price for ENS bandwidth, expressed in GB.

[0037] Step 2: To solve the long-term optimization problem described in Step 1, which involves the cost and request latency throughout the entire lifecycle of the data object, the problem is transformed, including the following steps: Step 2.1: Introduce a virtual queue This indicates the cumulative latency value exceeding the upper limit of acceptable latency for the end user as of the current time slot. The update method is as follows: (9) Step 2.2: To ensure the delay constraint in the original integer programming problem is guaranteed, the queue needs to be guaranteed. To assess queue stability, the Lyapunov function is introduced to measure queue stability: (10) The queue is guaranteed by optimizing the difference between the Lyapunov functions of two adjacent time slots. The stability of the Lyapunov function depends on the upper bound of the difference between two adjacent time slots, namely: (11) Therefore, the queue can be guaranteed by optimizing the upper bound of the difference between the Lyapunov functions of two adjacent time slots. The stability of the problem ensures that the latency-related constraints in the original problem are guaranteed. Up to this step, the long-term constraint of the request latency of the required data object throughout its entire lifecycle in the original problem is transformed into a short-term optimization problem. By minimizing this problem, the long-term constraints in the original problem can be guaranteed.

[0038] Step 2.3: Decompose the optimization objectives for the entire lifecycle to obtain... The cost optimization objective for a single time slot is combined with the delay optimization objective obtained in step 2.2 to construct a new optimization objective, forming... There are several short-run optimization problems, each of which is formulated as follows: (12) By solving the above integer programming problem, the edge replica placement strategy and cloud storage category selection strategy that satisfy the latency constraints of each time slot terminal user's request can be obtained.

[0039] Step 3: Enable logging in the edge storage instance to record the time when each data replica in each edge region is requested. Statistical analysis of each log entry will yield the number of requests for each time slot.

[0040] Step 4: Based on the request count of each edge region of the current time slot obtained in Step 3, approximate the short-term optimization problem of the current time slot obtained in Step 2, including the following steps: Step 4.1: Calculate the current edge replica placement strategy and cloud storage types The objective value of the short-term optimization problem in the current time slot .

[0041] Step 4.2: Randomly select an edge region, modify its placement strategy, and obtain a new edge copy placement strategy. ,calculate The optimal value of the short-term optimization problem in the current time slot And record the storage category of the cloud replica corresponding to the obtained optimal value. .

[0042] Step 4.3: Based on the target values ​​before and after the edge region placement strategy modification, calculate the value between 0 and 1: (13) in This indicates user-defined parameters.

[0043] Step 4.4: with The probability of accepting this change in the edge copy placement strategy is... , ;by The probability of rejecting this change in the edge copy placement strategy is high. , .

[0044] Step 4.5: Repeat steps 4.1 to 4.4 one thousand times or for ten consecutive times. If the target value increases by less than 0.1%, stop the iteration.

[0045] Step 5: Based on the results obtained in Step 4 and Adjust the storage type for replicas in edge regions and replicas in the cloud. Traverse... For each term, if it is 0 and If the corresponding item is 1, then the copy in the corresponding edge region will be deleted; if it is 1 and If the corresponding item is 0, then select the nearest edge region with the lowest latency that contains a replica to download the data, and create a new replica in the corresponding edge region; Step 6: Repeat steps 3-5 in each subsequent time slot until the data object's lifecycle ends.

[0046] This embodiment uses well-known cloud service providers Amazon Web Services and Alibaba Cloud as examples to detail the implementation steps of the technical solution proposed in this invention, and is representative.

[0047] Example 2: Embodiment 2 of the present invention provides a data cloud-edge collaborative management system based on public cloud storage services, such as... Figure 2 The diagram shows the architecture of this system. This system involves cloud storage services and edge storage services. Each storage instance in the edge region can directly obtain data from the cloud storage service. Storage instances in adjacent edge regions can communicate and transmit data. This embodiment takes Amazon Cloud S3 service and Alibaba Cloud ENS service as examples to build a cloud-edge collaborative storage framework and deploy the system in the ENS service. This system includes four modules: price crawler module, request collection module, communication module, and scheduling optimization module.

[0048] The price crawler module runs on all edge storage instances and is configured to periodically retrieve the prices of S3 services and ENS services for all regions in each time slot. This module uses a crawler program to download, parse, and store the price display page in a local SQLite database on each edge storage instance.

[0049] The request collection module runs on all edge storage instances and is configured to periodically collect the number of requests for the cloud replicas and their corresponding replicas in the edge region for each time slot. In order to analyze the number of requests for the cloud replicas, the logging function of Amazon S3 service is enabled and the logs are saved as separate data objects. This module downloads the log objects and performs statistical analysis on the request logs of each data object to obtain the number of times each data object's replica in the cloud was requested. Similarly, for the edge storage instance where this module is located, its request logs are analyzed to count the number of requests within the current time slot, thus obtaining the number of requests for data objects within that edge region. For replicas in other edge regions, the number of requests for each edge region is obtained through the communication module. Finally, the number of requests for the replicas in the cloud and for each edge region is stored in the SQLite database of the edge storage instance where this module is running.

[0050] The communication module runs in all edge storage instances and is configured to transmit information between different edge regions via sockets. The transmitted information includes the number of times other edge storage requests a certain data object, whether other edge regions have data copies, and the edge copy placement strategy. The communication module, which runs on the storage instance where the scheduling optimization module is deployed, receives the number of requests from storage instances in other edge regions and whether there are replicas, and sends the replica placement strategy of the scheduling optimization module to other edge regions. The communication module running on storage instances that do not have a scheduling optimization module deployed is responsible for sending their replica status and request information to storage instances that have a scheduling optimization module deployed, and for receiving replica placement policies. At the same time, the module will also update the replica status on the storage instance according to the received replica placement policy, that is, whether to create a new replica or destroy an old replica.

[0051] The scheduling optimization module runs the data cloud-edge collaborative management method based on public cloud storage services proposed in this invention, including a time slot target calculation unit and an objective function solution unit, configured as follows: The system retrieves request counts, storage service prices, and replica liveness status for all edge regions from the database. It then executes a short-term optimization algorithm for each time slot and performs an approximate solution. Based on the solution results, it adjusts the replicas in the cloud and the current edge region. The replica placement strategies for other edge regions are transmitted to the corresponding edge storage instances via the communication module. The adjustment of replica placement strategies for edge storage instances is achieved through file operations, while the adjustment of storage categories for cloud replicas is achieved through the cloud service provider's SDK.

[0052] It is understood that the methods of obtaining and the permissions to use the data described in this embodiment are all legal, such as data obtained by web crawlers.

[0053] Example 3: Embodiment 3 of the present invention provides a data cloud-edge collaborative management system based on public cloud storage services, comprising: The time slot target calculation unit is configured as follows: For a data object, with whether a replica is created in the edge region and the storage type of the replica in the cloud server as decision variables, the portion of the average latency of the current time slot that exceeds the upper limit of the latency acceptable to the end user is calculated, and the cumulative latency exceeding the upper limit of the latency acceptable to the end user up to the current time slot is used as the weight to obtain the first variable; Calculate the cost generated in the current time slot, using user-defined parameters as weights to obtain a second variable, and use the minimum weighted sum of the first and second variables as the objective function; The objective function solving unit is configured as follows: Based on the number of requests in each edge region of the current time slot, the objective function is solved to obtain the placement strategy of this data object in the edge region and its storage category on the cloud server for the current time slot.

[0054] The working method of each unit of the system is the same as the data cloud-edge collaborative management method based on public cloud storage service provided in Embodiment 1, and will not be repeated here.

[0055] Example 4: Embodiment 4 of the present invention provides a computer-readable storage medium having a program stored thereon. When the program is executed by a processor, it implements the steps in the data cloud-edge collaborative management method based on public cloud storage services as described in Embodiment 1 of the present invention.

[0056] Example 5: Embodiment 5 of the present invention provides an electronic device, including a memory, a processor, and a program stored in the memory and executable on the processor. When the processor executes the program, it implements the steps in the data cloud-edge collaborative management method based on public cloud storage services as described in Embodiment 1 of the present invention.

[0057] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of hardware embodiments, software embodiments, or embodiments combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage and optical storage) containing computer-usable program code.

[0058] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0059] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0060] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0061] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0062] The above description is merely a preferred embodiment of the present invention and is not intended to limit the invention. Various modifications and variations can be made to the present invention by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A data cloud edge collaboration management method based on a public cloud storage service, characterized in that, The process includes the following: For a data object, with whether a replica is created in the edge region and the storage type of the replica in the cloud server as decision variables, the portion of the average latency of the current time slot that exceeds the upper limit of the latency acceptable to the end user is calculated, and the cumulative latency exceeding the upper limit of the latency acceptable to the end user up to the current time slot is used as the weight to obtain the first variable; Calculate the cost generated in the current time slot, using user-defined parameters as weights to obtain a second variable, and use the minimum weighted sum of the first and second variables as the objective function; Based on the number of requests in each edge region of the current time slot, the objective function is solved to obtain the placement strategy of this data object in the edge region and its storage category on the cloud server for the current time slot. As of now, the cumulative latency exceeding the upper limit of acceptable latency for end users in the current time slot is: ； wherein, is the cumulative value of delay exceeding the upper bound of the terminal user acceptable delay until the previous time slot, denotes the decision variable of the time slot, and the average delay of all requests in the current time slot; denotes the upper bound of the average delay acceptable to the terminal user, is a dimensional variable, each item representing whether a replica is created in an edge area, is the storage category of the data replica in the cloud. The objective function is solved based on the number of requests for each edge region in the current time slot, including: Calculate the objective value of the short-term optimization problem for the current time slot under the current edge replica placement strategy and cloud storage type. ; Randomly select an edge region, modify its placement strategy, and obtain a new edge copy placement strategy. ,calculate The optimal value of the short-term optimization problem in the current time slot And record the storage category of the cloud replica corresponding to the obtained optimal value. ; Based on the target values ​​before and after the edge region placement strategy modification, calculate the value between 0 and 1: ,in This indicates user-defined parameters; by The probability of accepting this change in the edge copy placement strategy is... , ;by The probability of rejecting this change in the edge copy placement strategy is high. , ; If the target value increases by less than 0.1% after repeating the above steps a set number of times, stop the iteration. According to the obtained and Adjust the storage type of the replicas in the edge region and the replicas in the cloud, and traverse... For each term, if it is 0 and If the corresponding item is 1, then the copy in the corresponding edge region will be deleted; if it is 1 and If the corresponding item is 0, then select the nearest edge region with the lowest latency that contains a replica to download the data, and create a new replica in the corresponding edge region.

2. The data cloud-edge collaborative management method based on public cloud storage services as described in claim 1, characterized in that, Throughout the entire lifecycle of this data object, the objective function is optimized and solved sequentially for each time slot to obtain the placement strategy of this data object in the edge region and the storage category on the cloud server for each time slot.

3. The data cloud-edge collaborative management method based on public cloud storage services as described in claim 1 or 2, characterized in that, Each time slot of a data object is a sub-time period within the entire lifecycle. Within the entire lifecycle, the goal is to minimize the cost of using cloud-edge storage, while the upper limit of the average latency of all end-user requests is a constraint. Long-term integer planning is performed across the entire lifecycle, using whether to create replicas in edge regions and the storage category of replicas in the cloud as decision variables. The long-term optimization problem spanning the entire lifecycle is transformed into a short-term optimization problem within each time slot by using the optimization objective and constraints of a time slot under a long-term integer programming over the entire lifecycle. The objective function of each time slot is a weighted sum of the cost and unmet delays of each time slot.

4. The data cloud-edge collaborative management method based on public cloud storage services as described in claim 1, characterized in that, Whether to create a replica in the edge region is the primary decision variable, and each variable takes a value of 0 or 1. dimensional vector, This indicates that no data copy was created for the corresponding edge region. This indicates that a data copy exists in the corresponding edge region; In cloud servers, the storage category of replicas is the second decision variable, where 0 represents standard storage, 1 represents low-frequency storage, and 2 represents archive storage.

5. A data cloud-edge collaborative management system based on public cloud storage services, characterized in that, include: The module includes a price crawler module, a request collection module, a communication module, and a scheduling optimization module. The price crawler module is configured to retrieve price data for edge cloud services and cloud services for each time slot. The request collection module is configured to: obtain the number of requests for each replica in the time slot cloud service and the replicas in its edge region; The communication module is configured to run in all edge storage instances and transmit information between different edge regions. The information transmitted includes the number of times the edge storage requests a certain data object, whether there are data replicas in the edge region, and the edge replica placement strategy. The scheduling optimization module is configured to execute the steps of the data cloud-edge collaborative management method based on public cloud storage services as described in any one of claims 1-4.

6. A data cloud-edge collaborative management system based on public cloud storage services, characterized in that, The data cloud-edge collaborative management method based on public cloud storage services according to any one of claims 1-4 includes: The time slot target calculation unit is configured as follows: For a data object, with whether a replica is created in the edge region and the storage type of the replica in the cloud server as decision variables, the portion of the average latency of the current time slot that exceeds the upper limit of the latency acceptable to the end user is calculated, and the cumulative latency exceeding the upper limit of the latency acceptable to the end user up to the current time slot is used as the weight to obtain the first variable; Calculate the cost generated in the current time slot, using user-defined parameters as weights to obtain a second variable, and use the minimum weighted sum of the first and second variables as the objective function; The objective function solving unit is configured as follows: Based on the number of requests in each edge region of the current time slot, the objective function is solved to obtain the placement strategy of this data object in the edge region and its storage category on the cloud server for the current time slot.

7. A computer-readable storage medium having a program stored thereon, characterized in that, When executed by the processor, the program implements the steps in the data cloud-edge collaborative management method based on public cloud storage services as described in any one of claims 1-4.

8. An electronic device comprising a memory, a processor, and a program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps in the data cloud-edge collaborative management method based on public cloud storage services as described in any one of claims 1-4.

Images