A multi-cloud storage processing method, device, equipment and storage medium

Abstract

The application provides a multi-cloud storage processing method, device and equipment and storage medium, and belongs to the technical field of multi-cloud storage. The method comprises the following steps: obtaining target data to be stored, and dividing the target data into multiple data segments; performing a multi-cloud storage segmented online algorithm strategy to determine target storage nodes of the multiple data segments corresponding to the target data, including performing an allocation strategy and performing an allocation adjustment strategy; wherein the allocation strategy is used to determine the target storage nodes of the multiple data segments corresponding to the target data; the allocation adjustment strategy is used to adjust the target storage nodes of the multiple data segments of the target data according to a storage balance point when there is any access request, and the storage balance point is used to represent a balance relationship between the number of access requests and the storage cost of the target data. The application aims to reduce the storage cost.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the technical field of multi-cloud storage, in particular, to a multi-cloud storage processing method, device, equipment and storage medium. BACKGROUND

[0002] With the maturity and popularity of cloud computing technology, more and more enterprises and individuals begin to store data in the cloud to obtain higher reliability, flexibility and scalability, and major enterprises also begin to adopt multi-cloud strategy to reduce data storage risks and improve data storage flexibility by storing data on multiple cloud platforms.

[0003] With the development of artificial intelligence and big data analysis technology, multi-cloud storage also begins to integrate intelligent functions to help users better manage and analyze stored data; multi-cloud storage strategy has many advantages, not only can allocate storage resources between different cloud storage service providers according to storage needs to better meet business needs and respond to sudden traffic, but also can optimize data access speed and performance and improve application response capability by dispersing storage load on different cloud storage platforms.

[0004] In the current digital construction, multi-cloud storage strategy has broad development space in terms of technological innovation, security control, intelligence and ecological system construction, but as the storage demand of data continues to surge, combined with the fact that cloud service providers usually charge according to the usage amount of storage, high storage costs often occur in the multi-cloud environment, and in the multi-cloud environment, cloud service providers may adjust data storage pricing strategies from time to time, which may lead to instability of storage costs, resulting in high storage costs. SUMMARY

[0005] Embodiments of the present application provide a multi-cloud storage processing method, device, equipment and storage medium, aiming to reduce storage costs.

[0006] In a first aspect, embodiments of the present application provide a multi-cloud storage processing method, which comprises:

[0007] Obtaining target data to be stored, and dividing the target data into multiple data segments;

[0008] Executing a multi-cloud storage segmented online algorithm strategy to determine the target storage nodes of each of the multiple data segments corresponding to the target data, including executing an allocation strategy and executing an allocation adjustment strategy;

[0009] The allocation strategy is used to determine target storage nodes of the plurality of data segments of the target data; and the allocation adjustment strategy is used to adjust the target storage nodes of the plurality of data segments of the target data according to a storage balance point when any access request exists, the storage balance point being used to represent a balance relationship between the number of access requests and the storage cost of the target data.

[0010] Optionally, the target data to be stored is obtained, and the target data is divided into a plurality of data segments, including:

[0011] The target data to be stored is obtained.

[0012] The target data is divided into a plurality of data segments according to a total amount of data of the target data, current usage information of each storage node, and a storage characteristic of the target data.

[0013] Optionally, the allocation strategy is executed, including:

[0014] The usage information of each storage node and the remaining storage capacity of each storage node are obtained.

[0015] For any data segment of the target data, when the usage information of any storage node represents an idle state and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, the storage node is taken as a target storage node of the data segment.

[0016] When the usage information of any storage node represents a saturation state, or the remaining storage capacity of the storage node is less than the data capacity of the data segment, a next storage node is traversed until the target storage node of the data segment is determined.

[0017] Optionally, the allocation adjustment strategy is executed, including:

[0018] The remaining storage capacity of each storage node at a current time is obtained in real time, and each storage node is sorted and added in a storage node queue according to the size of the remaining storage capacity.

[0019] When any access request exists, the target storage nodes of the plurality of data segments of the target data are adjusted in the storage node queue according to the number of access requests of the target data and the storage balance point.

[0020] Optionally, adjusting the target storage nodes of the plurality of data segments of the target data in the storage node queue according to the number of access requests of the target data and the storage balance point includes:

[0021] When the number of access requests of the target data is less than or equal to the storage balance point, any data segment of the target data is sequentially stored in the storage nodes in the storage node queue.

[0022] wherein, when the remaining storage capacity of the current storage node in the storage node queue is less than or equal to the data capacity of the current data segment of the target data, determining the to-be-migrated data in the current storage node according to the dynamic flexible window, and migrating the to-be-migrated data to the storage node at the tail of the storage node queue until the remaining storage capacity of the current storage node is greater than the data capacity of the current data segment, and taking the current storage node as the target storage node of the current data segment;

[0023] When the number of access requests of the target data is greater than the storage balance point, re-executing the multi-cloud storage segmentation online algorithm strategy for the target data.

[0024] Optionally, the allocation adjustment strategy further comprises:

[0025] calculating the storage balance point;

[0026] The calculation formula of the storage balance point is:

[0027]

[0028] wherein, Q is the target data; is the data capacity of the jth data segment of the target data; is the storage unit price of the storage node i; is the storage duration of the target data; is the access request processing cost unit price; n is the number of storage nodes; and m is the number of data segments of the target data.

[0029] Optionally, the allocation adjustment strategy further comprises:

[0030] calculating the size of the dynamic flexible window, the size of the dynamic flexible window representing the size of the data amount stored per unit time;

[0031] The calculation formula of the size of the dynamic flexible window is:

[0032]

[0033] wherein, is the size of the dynamic flexible window; T is the last storage period; is the number of access requests of the target data; is the data capacity of the jth data segment of the target data; is the access request processing cost unit price; is the sensitivity coefficient.

[0034] In a second aspect, the embodiments of the present application provide a processing device for multi-cloud storage, the device comprising:

[0035] an acquisition module configured to acquire target data to be stored and divide the target data into a plurality of data segments;

[0036] a storage module configured to execute a multi-cloud storage segmented online algorithm strategy to determine target storage nodes for the plurality of data segments corresponding to the target data, including executing an allocation strategy and executing an allocation adjustment strategy;

[0037] The allocation strategy is configured to determine the target storage nodes for the plurality of data segments corresponding to the target data, and the allocation adjustment strategy is configured to adjust the target storage nodes for the plurality of data segments of the target data according to a storage balance point when there is any access request, the storage balance point being configured to represent a balance relationship between the number of access requests and the storage cost of the target data.

[0038] In a third aspect, the embodiments of the present application provide a computer device, comprising at least one processor and a memory, the memory storing a computer program capable of running on the processor, wherein the processor executes the computer program to execute the processing method for multi-cloud storage according to the first aspect of the embodiments.

[0039] In a fourth aspect, the embodiments of the present application provide a non-volatile readable storage medium, the non-volatile readable storage medium storing a computer program, wherein the computer program is executed by a processor to execute the processing method for multi-cloud storage according to the first aspect of the embodiments.

[0040] Advantages:

[0041] The target data to be stored is acquired and divided into a plurality of data segments, a multi-cloud storage segmented online algorithm strategy is executed to determine target storage nodes for the plurality of data segments corresponding to the target data, including executing an allocation strategy and executing an allocation adjustment strategy, wherein the allocation strategy is configured to determine the target storage nodes for the plurality of data segments corresponding to the target data, and the allocation adjustment strategy is configured to adjust the target storage nodes for the plurality of data segments of the target data according to a storage balance point when there is any access request, the storage balance point being configured to represent a balance relationship between the number of access requests and the storage cost of the target data.

[0042] The method provided in the embodiments can divide the target data into a plurality of data segments according to a multi-cloud storage segmented online algorithm when storing the target data, and then store the plurality of data segments in corresponding target storage nodes, and can dynamically adjust the target storage nodes of each data segment according to the number of access requests and the storage balance point when accessing the stored target data, thereby reducing the storage cost of data. Attached Figure Description

[0043] To more clearly illustrate the technical solutions of the embodiments of this application, the drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0044] Figure 1 This is a flowchart of the steps of a multi-cloud storage processing method proposed in an embodiment of this application;

[0045] Figure 2 This is a flowchart illustrating the steps of implementing an allocation strategy according to an embodiment of this application;

[0046] Figure 3 This is a flowchart illustrating the steps of an execution allocation adjustment strategy provided in an embodiment of this application;

[0047] Figure 4 This is an execution flowchart of a multi-cloud storage processing method proposed in an embodiment of this application;

[0048] Figure 5 This is a functional block diagram of a multi-cloud storage processing device proposed in an embodiment of this application;

[0049] Figure 6 This is a schematic diagram of a computer device provided in an embodiment of this application;

[0050] Figure 7 This is a schematic diagram of a non-volatile readable storage medium proposed in an embodiment of this application. Detailed Implementation

[0051] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0052] Segmented online algorithms are algorithms that can respond in real time to dynamic changes in data streams when processing service requests. They can effectively handle situations where different characteristic segments appear in the data stream when the data distribution suddenly changes or new data patterns emerge. Segmented online algorithms can make real-time analysis and effective decisions in various fields such as industrial production optimization and intelligent transportation systems where data streams are dynamically changing.

[0053] Multi-cloud storage refers to the practice of storing data and resources across multiple cloud service providers by cloud tenants. This approach allows users to flexibly allocate storage resources among different cloud service providers based on their storage needs, thereby better meeting business requirements. By storing data on different cloud platforms, users can achieve data backup and disaster recovery.

[0054] Storage equilibrium point refers to the critical point calculated for the storage cost of different feature segments during the process of using online algorithms to handle data streams. When calculating the storage equilibrium point, the response speed of online algorithms and the accuracy of processing results need to be considered to meet the storage needs of different application scenarios. The determination of the equilibrium point also needs to take into account factors such as the characteristics of data streams, system resource usage, and algorithm complexity.

[0055] Dynamic sliding window is a commonly used technique in handling data streams. Unlike traditional sliding windows, dynamic sliding windows can dynamically adjust the size or position of the window based on the characteristics of the data stream to adapt to real-time changes in the data stream. It is commonly used in real-time data analysis, pattern recognition, and stream computing, and can effectively handle changing data streams and extract useful information or features.

[0056] With the maturity and popularity of cloud computing technology, more and more enterprises and individuals are starting to store data in the cloud to achieve higher reliability, flexibility, and scalability. Major enterprises are also adopting multi-cloud strategies by storing data on multiple cloud platforms to reduce data storage risks and improve data storage flexibility. With the development of artificial intelligence and big data analysis technology, multi-cloud storage is also integrating intelligent functions to help users better manage and analyze stored data.

[0057] Multi-cloud storage strategy has many advantages. Not only can it flexibly allocate storage resources among different cloud storage service providers based on storage needs to better meet business requirements and handle sudden traffic, but it can also optimize data access speed and performance by distributing storage load across different cloud storage platforms, improving application response capabilities.

[0058] In today's digital construction, multi-cloud storage strategy has broad development space in terms of technological innovation, security control, intelligence, and ecosystem construction. However, with the increasing demand for data storage, cloud service providers usually charge based on storage usage, resulting in high storage costs in a multi-cloud environment.

[0059] By optimizing multi-cloud storage costs, enterprises can not only reduce operating costs and effectively utilize storage resources, but also maintain a competitive advantage in the market and achieve long-term sustainable development. Therefore, how to further optimize storage costs in a multi-cloud environment is a problem worth further study.

[0060] In a multi-cloud environment, a cloud service provider can adjust the data storage pricing strategy from time to time, which can cause instability of storage costs and unnecessary waste of storage resources.

[0061] Therefore, the embodiments of the present application provide a multi-cloud storage processing method, which can meet the storage needs of users while reducing storage costs.

[0062] Referring to Figure 1 , a step flowchart of a multi-cloud storage processing method in the embodiments of the present application is shown, the method is applied to the user side, and the method can specifically include the following steps:

[0063] S101: Obtain target data to be stored, and divide the target data into multiple data segments.

[0064] In actual implementation, a user will store any data in a multi-cloud storage service according to the actual storage business needs, the multi-cloud storage service includes multiple storage nodes provided by different cloud service providers, and different storage nodes provided by the same cloud service provider can have different storage costs because of the different performances of the storage nodes.

[0065] Because the present method is based on a segmented online algorithm, the segmented online algorithm is a commonly used algorithm in data stream processing, mainly used for processing dynamic data sets, the algorithm processes data streams in segments, and each segment is calculated independently, thereby realizing real-time processing and analysis of data streams.

[0066] Therefore, when any user stores target data in a multi-cloud storage service, the data sequence of the target data needs to be divided first to obtain multiple data segments corresponding to the target data; for example, the target data is denoted as , the target data is divided into m data segments, and any data segment is denoted as , j∈[1, m].

[0067] In a feasible implementation, in the process of obtaining target data to be stored and dividing the target data into multiple data segments, the target data to be stored is obtained first, and then the target data is divided into multiple data segments according to the total amount of data of the target data, the current usage information of each storage node, and the storage characteristics of the target data.

[0068] The usage information corresponding to each storage node represents the storage state of the storage node; for example, the i-th storage node in n storage nodes in the multi-cloud storage service is denoted as , and the usage information of any storage node is denoted as When =1, the storage node is in an idle state, and normal storage process can be performed; when =0, the storage node is in a saturated state, and normal storage process cannot be performed.

[0069] The storage characteristics of any data include the access frequency of the data, etc.

[0070] In actual implementation, the target data can be divided into multiple data segments in combination with the total amount of data of the target data, the current usage information of each storage node, and the storage characteristics of the target data.

[0071] The specific division rule of the target data can be customized according to actual storage requirements. For example, a division rule can be preset as follows: determining the division weights of the total amount of data of the data, the current usage information of each storage node, and the storage characteristics of the data, respectively, and then determining the number of corresponding data segments according to the range of the weighted results. In actual implementation, other division rules can also be used, and the present embodiment is not limited thereto.

[0072] S102: performing a multi-cloud storage segmentation online algorithm strategy to determine the target storage nodes of the multiple data segments corresponding to the target data, including performing an allocation strategy and performing an allocation adjustment strategy.

[0073] The allocation strategy is used to determine the target storage nodes of the multiple data segments corresponding to the target data.

[0074] The allocation adjustment strategy is used to adjust the target storage nodes of the multiple data segments of the target data according to a storage balance point when there is any access request, and the storage balance point is used to represent the balance relationship between the number of access requests and the storage cost of the target data.

[0075] Referring to Figure 2 , a step flowchart of performing the allocation strategy provided by the present embodiment is shown. In a feasible implementation manner, the process of performing the allocation strategy can include the following steps:

[0076] A1: obtaining the usage information of each storage node and the remaining storage capacity of each storage node.

[0077] Specifically, at any time, the usage information of each storage node in the last storage period can be collected, and the remaining storage capacity of each storage node is recorded respectively. The remaining storage capacity of the i-th storage node is . ​​

[0078] The storage period is configured by the user, for example, the storage period can be one month or one year, etc., which can be defined according to the needs in the actual implementation process.

[0079] A2: For any data segment of the target data, when the usage information of any storage node represents an idle state and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, the storage node is taken as the target storage node of the data segment.

[0080] After collecting the usage information of each storage in the last storage period and recording the remaining storage capacity of each storage node, it can be determined whether any storage node is in an idle state, and the remaining storage capacity of the storage node in the idle state, so as to determine whether any data segment of the target data can be stored in the storage node.

[0081] In the actual implementation process, since the target data corresponds to multiple data segments, each data segment can be traversed in order, and the target storage node corresponding to the data segment is determined.

[0082] For any data segment of the target node, the usage information and the remaining storage capacity of each node are traversed. When a storage node is in an idle state, it represents that the normal storage process can be performed, and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, which indicates that the data segment can be stored in the storage node, and then the storage node is taken as the target storage node of the data segment.

[0083] A3: When the usage information of any storage node represents a saturated state, or the remaining storage capacity of the storage node is less than the data capacity of the data segment, the next storage node is traversed until the target storage node of the data segment is determined.

[0084] For any data segment of the target node, when a storage node is in a saturated state, it represents that the storage node cannot perform a normal storage process, and then the storage node is skipped, and the next storage node is continued to be determined whether it can be the target storage node of the data segment.

[0085] Or, when a storage node is in an idle state, i.e., the storage node can perform a normal storage process, but the remaining storage capacity of the storage node is less than the data capacity of the data segment, the data segment cannot be stored in the storage node, and then the storage node is skipped, and the next storage node is continued to be determined whether it can be the target storage node of the data segment.

[0086] When a storage node is found to be in an idle state and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, the storage node is taken as a target storage node of the data segment, and then the target storage node of the next data segment of the target data is determined.

[0087] When all data segments of the target data are determined to correspond to the target storage nodes, the data segments are stored in the target storage nodes, and the remaining storage capacities of the target storage nodes after storing the corresponding data segments are recorded, so that the target data to be stored is preliminarily stored in the multi-cloud storage.

[0088] After the allocation strategy determines the target storage nodes for all data segments of the target data, the storage nodes of any data segment are not fixed. To reduce the storage cost of data, the method provided in the embodiment further performs an allocation adjustment strategy. In the process of a subsequent access request for the target data, the storage positions of any data segment are adjusted according to the storage balance point, so as to minimize the storage cost of data.

[0089] Referring to Figure 3 , a step flowchart of performing the allocation adjustment strategy provided in the embodiment is shown. In a feasible implementation manner, the process of performing the allocation adjustment strategy can include the following steps.

[0090] B1: Real-time acquisition of the remaining storage capacities of all storage nodes at the current time, and sorting of the storage nodes according to the sizes of the remaining storage capacities and adding the sorted storage nodes in a storage node queue.

[0091] Real-time updating of the sizes of the remaining storage capacities of all storage nodes, and sorting of all storage nodes in ascending order of the sizes of the remaining storage capacities and adding the sorted storage nodes in the storage node queue.

[0092] In the multi-cloud storage, there are generally hot storage layers, warm storage layers, and cold storage layers. The storage nodes in the hot storage layers are generally nodes with fast storage speed but small storage capacity, which can quickly respond to any access request of a user for data. The storage nodes in the cold storage layers are generally nodes with low storage speed but large storage capacity.

[0093] Therefore, after sorting according to the sizes of the remaining storage capacities, the nodes at the head of the storage node queue are mostly nodes in the hot storage layers, and the nodes at the tail of the storage node queue are mostly nodes in the cold storage layers with large storage capacity.

[0094] B2: When there is any access request, the target storage nodes of the data segments of the target data are adjusted in the storage node queue according to the access request number of the target data and the storage balance point.

[0095] When a user also makes a request for accessing the stored target data after storing the target data, the method can adjust the storage locations of the plurality of data segments of the target data according to the access to the target data.

[0096] In actual implementation, the storage balance point is calculated.

[0097] According to n storage nodes, d storage balance points are set , k∈[1, d], n=d+1.

[0098] In the storage of the plurality of data segments of the target data, the total storage cost Z of the target data includes the sum of the data storage cost X and the data processing cost Y and other additional fees , and the calculation formula of the total storage cost Z is:

[0099]

[0100] Specifically,

[0101]

[0102] wherein Z is the total storage cost; is the data capacity of the jth data segment of the target data; is the storage unit price of the storage node i; is the storage duration of the target data; is the number of access requests; is the access request processing cost unit price; n is the number of storage nodes; and m is the number of data segments of the target data. is the additional fee.

[0103] The additional fee includes other storage fees in multi-cloud storage. For example, the additional fee includes storage discounts.

[0104] Further, according to the calculation formula of the total storage cost, the calculation formula of the storage balance point is:

[0105]

[0106] wherein Q is the target data; is the data capacity of the jth data segment of the target data; is the storage unit price of the storage node i; is the storage duration of the target data; is the access request processing cost unit price; n is the number of storage nodes; and m is the number of data segments of the target data.

[0107] Then according to the access request number of the target data and the storage balance point, the process of adjusting the target storage node of the data segment of the target data in the storage node queue includes two cases.

[0108] The first case: when the access request number of the target data is less than or equal to the storage balance point, any data segment of the target data is stored in the storage node in the storage node queue in turn.

[0109] That is, when the access request number ≤ storage balance point , the data segment of the target data can be adjusted according to the order of each storage node in the storage node queue.

[0110] In the actual implementation process, each storage node in the storage node queue is traversed for any data segment.

[0111] When the remaining storage capacity of the current storage node in the storage node queue is less than or equal to the data capacity of the current data segment of the target data, the data to be migrated in the current storage node is determined according to the dynamic flexible window, and the data to be migrated is migrated to the storage node at the tail of the storage node queue.

[0112] Specifically, the embodiment also provides a formula for calculating the size of the dynamic flexible window, which represents the size of the data stored per unit time.

[0113] For example, from time t-T+1, the number of access requests in the last storage period T is traversed, and the calculation formula of the data processing cost Y in the last storage period T is combined, that is:

[0114]

[0115] The size calculation formula of the dynamic flexible window is:

[0116]

[0117] Wherein, wherein, the size of the dynamic flexible window; T is the last storage period; the access request number of the target data; the data capacity of the jth data segment of the target data; the data capacity of the jth data segment of the target data; the sensitivity coefficient.

[0118] The sensitivity system is a dynamic elastic window, which can be set according to the actual application requirements according to the size of the change of the storage load, and the larger the storage load is, the larger the window size is.

[0119] When the remaining storage capacity of the current storage node in the storage node queue is less than or equal to the data capacity of the current data segment of the target data, the data not accessed in the last storage period T can be regarded as the to-be-migrated data, and then the number of the to-be-migrated data is determined according to the size of the dynamic elastic window, and the to-be-migrated data is migrated to the storage node at the tail of the storage node queue.

[0120] Until the remaining storage capacity of the current storage node is greater than the data capacity of the current data segment, the current storage node is regarded as the target storage node of the current data segment.

[0121] In the actual implementation process, if the to-be-migrated data is migrated to the storage node at the tail of the storage node queue through the dynamic elastic window, and the remaining storage capacity of the current storage node is still less than the current data segment, the next storage node of the storage node queue can be traversed to determine the adjusted target storage node corresponding to the data segment.

[0122] Through the dynamic elastic window, the unaccessed data can be migrated to the storage node such as the cold storage layer according to the access request condition of the current target data when the access request times ≤ storage balance point of the target data, and the data segment of the target data is left out of the storage space, and the size of the dynamic elastic window can also be adjusted according to the access request condition of the data segment of the current target data.

[0123] The second case: when the access request times of the target data is greater than the storage balance point, the multi-cloud storage segmentation online algorithm strategy is re-executed for the target data, that is, when the access request times is greater than the storage balance point , the storage of the target data is unreasonable, which may cause a large storage cost, at this time, the target data is re-divided into multiple data segments, and the multi-cloud storage segmentation online algorithm strategy is executed again to determine the target storage node.

[0124] Referring to Figure 4 , an execution flowchart of a processing method of multi-cloud storage according to an embodiment of the present application is shown, and in a feasible implementation manner, the method can include the following steps:

[0125] S1: The obtained target data is divided into multiple data segments.

[0126] For example, the target data is denoted as , and the target data is divided into m data segments, any data segment is denoted as , the data capacity of any data segment j is , j ∈ [1, m].

[0127] S2: execute the allocation strategy to allocate the multiple data segments to the respective corresponding target storage nodes.

[0128] Collect the usage information of each storage node in the last storage period T, and record the remaining storage capacity of the i-th storage node . .

[0129] When the remaining storage capacity of the i-th storage node is greater than or equal to the data capacity of any data segment j , and when =1, the storage node is used as the target storage node of the data segment; When the remaining storage capacity of the i-th storage node

[0130] is less than the data capacity of any data segment j , or when =1, the next storage node is traversed. S3: calculate the storage balance point and the dynamic elastic window.

[0131] Specifically, the calculation formula of the storage balance point is:

[0132]

[0133]

[0134] Wherein, Q is the target data; is the data capacity of the j-th data segment of the target data; is the storage unit price of the storage node i; is the storage duration of the target data; is the access request processing cost unit price; n is the number of storage nodes; m is the number of data segments of the target data.

[0135] The size calculation formula of the dynamic elastic window is:

[0136]

[0137] Wherein, wherein, is the size of the dynamic elastic window; T is the last storage period; is the number of access requests of the target data; ​​a data capacity of a jth data segment of the target data; a data capacity of a jth data segment of the target data; a sensitivity coefficient.

[0138] S4: Real-time acquisition of the residual storage capacity of each storage node at the current time, and sorting each storage node according to the size of the residual storage capacity and adding it in the storage node queue.

[0139] S5: When there is any access request, adjusting the target storage node of the data segment according to the access request number of the target data and the storage balance point.

[0140] S6: Judging whether the access request number is less than or equal to the storage balance point.

[0141] If the access request number is less than or equal to the storage balance point, step S7 is executed;

[0142] If the access request number is greater than the storage balance point, step S10 is executed.

[0143] S7: Judging whether the residual storage capacity of the current storage node is less than or equal to the data capacity of the current data segment.

[0144] If the residual storage capacity of the current storage node is less than or equal to the data capacity of the current data segment, step S8 is executed.

[0145] If the residual storage capacity of the current storage node is greater than the data capacity of the current data segment, step S9 is executed.

[0146] S8: Determining the to-be-migrated data in the current storage node according to the dynamic elastic window, and migrating the to-be-migrated data to the storage node at the tail of the storage node queue.

[0147] S9: Storing the current data segment in the current storage node.

[0148] Then, the next data segment is taken as the current data segment, and step S7 is continuously executed.

[0149] S10: Re-dividing the target data into multiple data segments, and executing S2.

[0150] The method provided by the embodiment can quickly respond to the changing storage demand, effectively reduce the data storage cost, and improve the performance and efficiency of data storage.

[0151] Then the plurality of data segments are respectively stored in corresponding target storage nodes, and when the stored target data is accessed afterwards, the target storage nodes of each data segment can be dynamically adjusted according to the access request number and the storage balance point, so as to reduce the storage cost of data.

[0152] In the multi-cloud environment, in order to meet the storage requirements while minimizing the storage cost, the method can not only provide a flexible, reliable and flexible multi-cloud storage service for cloud users, but also effectively reduce the cost of multi-cloud storage of cloud users; by introducing the storage balance point and the dynamic sliding window, when a new access request arrives, the data storage in the multi-cloud environment is adjusted and scheduled from the perspective of reducing the storage cost, so that the cloud user can monitor the storage of data in different cloud service providers in real time, and according to the storage load, the stored resources are allocated on multiple cloud platforms.

[0153] Reference Figure 5 , a functional module diagram of a multi-cloud storage processing device provided by an embodiment of the application is shown, and the device comprises:

[0154] The acquisition module 100 is configured to acquire target data to be stored, and divide the target data into a plurality of data segments.

[0155] The storage module 200 is configured to execute a multi-cloud storage segmentation online algorithm strategy, determine target storage nodes of the plurality of data segments corresponding to the target data, and comprises an allocation strategy and an allocation adjustment strategy.

[0156] The allocation strategy is configured to determine the target storage nodes of the plurality of data segments corresponding to the target data, and the allocation adjustment strategy is configured to adjust the target storage nodes of the plurality of data segments of the target data according to a storage balance point when there is any access request, and the storage balance point is configured to represent a balance relationship between the access request number and the storage cost of the target data.

[0157] Optionally, the acquisition module comprises:

[0158] The acquisition unit is configured to acquire target data to be stored.

[0159] The division unit is configured to divide the target data into a plurality of data segments according to a total amount of data of the target data, current use information of each storage node and a storage feature of the target data.

[0160] Optionally, the storage module comprises a first storage unit configured to execute the allocation strategy, comprising:

[0161] The first storage unit is configured to acquire current use information of each storage node and a remaining storage capacity of each storage node.

[0162] For any data segment of the target data, when the usage information of any storage node represents an idle state and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, the storage node is taken as a target storage node of the data segment;

[0163] When the usage information of any storage node represents a saturation state or the remaining storage capacity of the storage node is less than the data capacity of the data segment, the next storage node is traversed until the target storage node of the data segment is determined.

[0164] Optionally, the storage module comprises a second storage unit configured to execute an allocation adjustment strategy, comprising:

[0165] The remaining storage capacities of the storage nodes at the current time are acquired in real time, and the storage nodes are sorted according to the sizes of the remaining storage capacities and added in a storage node queue;

[0166] When there is any access request, the target storage nodes of the data segments of the target data are adjusted in the storage node queue according to the access request number of the target data and the storage balance point.

[0167] Optionally, the second storage unit is further configured to:

[0168] When the access request number of the target data is less than or equal to the storage balance point, any data segment of the target data is sequentially stored in the storage nodes in the storage node queue;

[0169] When the remaining storage capacity of the current storage node in the storage node queue is less than or equal to the data capacity of the current data segment of the target data, the data to be migrated in the current storage node is determined according to a dynamic elastic window, and the data to be migrated is migrated to the storage node at the tail of the storage node queue until the remaining storage capacity of the current storage node is greater than the data capacity of the current data segment, and the current storage node is taken as the target storage node of the current data segment;

[0170] When the access request number of the target data is greater than the storage balance point, the multi-cloud storage segmentation online algorithm strategy is re-executed for the target data.

[0171] With reference to Figure 6 , a schematic diagram of a computer device is shown, which comprises at least one processor 601 and a memory 602, and the memory 602 stores a computer program capable of running on the processor 601, wherein the processor 601 executes the computer program to execute the processing method of the multi-cloud storage according to the embodiments.

[0172] With reference toFigure 7 FIG. 7 shows a schematic diagram of a non-volatile readable storage medium according to an embodiment of the present application, the non-volatile readable storage medium 700 stores a computer program 701, wherein the computer program 701 is executed by a processor to perform the processing method of multi-cloud storage according to an embodiment of the present application.

[0173] Each of the embodiments in the present specification is described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other.

[0174] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, device or computer program product. Therefore, the embodiments of the present application can be in the form of a complete hardware embodiment, a complete software embodiment or an embodiment combining software and hardware aspects. Moreover, the embodiments of the present application can be in the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0175] The embodiments of the present application are described with reference to flowcharts and / or block diagrams according to the method, terminal device (system) and computer program product of the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of the flows and / or blocks in the flowcharts and / or block diagrams. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing terminal device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing terminal device produce a means for implementing the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 an apparatus that performs the functions specified in one block or multiple blocks.

[0176] These computer program instructions can also be stored in a computer readable storage medium that can guide the computer or other programmable data processing terminal device to work in a specific way, so that the instructions stored in the computer readable storage medium produce a product including instruction apparatus, which implements the functions specified in the flow Figure 1 one flow or multiple flows and / or blocks Figure 1 an apparatus that performs the functions specified in one block or multiple blocks.

[0177] These computer program instructions can also be loaded into a computer or other programmable data processing terminal device, so that a series of operation steps are performed on the computer or other programmable terminal device to produce a computer implemented process, so that the instructions executed on the computer or other programmable terminal device provide a means for implementing the functions specified in the flow Figure 1one or more processes and / or blocks Figure 1 the steps of a function specified in one or more blocks.

[0178] While the preferred embodiments of the application have been described, additional variations and modifications can be made to the embodiments by those skilled in the art once they learn of the basic inventive concepts. Therefore, the appended claims are intended to cover all such variations and modifications as fall within the scope of the application.

[0179] Finally, it should be noted that the terms "comprises", "comprising", or other variations thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can also include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises... a" does not, without more constraints, exclude the presence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0180] The principles and implementations of the present application have been described above with the specific examples. The above description of the embodiments is only intended to help understand the method of the present application and its core idea; meanwhile, for those skilled in the art, according to the idea of the present application, the specific implementation and application range can be changed. In summary, the content of the specification should not be understood as a limitation of the present application.

Claims

1. A method for processing multi-cloud storage, characterized in that, The method includes: Obtain the target data to be stored, and divide the target data into multiple data segments; The multi-cloud storage segmentation online algorithm strategy is executed to determine the target storage nodes for each of the multiple data segments corresponding to the target data, including executing allocation strategies and executing allocation adjustment strategies. The allocation strategy is used to determine the target storage nodes for each of the multiple data segments corresponding to the target data; the allocation adjustment strategy is used to adjust the target storage nodes of the multiple data segments of the target data according to the storage balance point when any access request exists, and the storage balance point is used to characterize the balance relationship between the number of access requests and the storage cost of the target data.

2. The method according to claim 1, characterized in that, Obtain the target data to be stored, and divide the target data into multiple data segments, including: Retrieve the target data to be stored; Based on the total amount of the target data, the current usage information of each storage node, and the storage characteristics of the target data, the target data is divided into multiple data segments.

3. The method according to claim 1, characterized in that, Execute the allocation strategy, including: Obtain the current usage information and remaining storage capacity of each storage node; For any data segment of the target data, when the usage information of any storage node indicates that it is in an idle state and the remaining storage capacity of the storage node is greater than or equal to the data capacity of the data segment, the storage node is taken as the target storage node of the data segment. When the usage information of any storage node indicates that it is saturated, or when the remaining storage capacity of the storage node is less than the data capacity of the data segment, the next storage node is traversed until the target storage node of the data segment is determined.

4. The method according to claim 3, characterized in that, Implement allocation adjustment strategies, including: The remaining storage capacity of each storage node is obtained in real time, and the storage nodes are sorted and added to the storage node queue according to the remaining storage capacity. When any access request exists, the target storage nodes for multiple data segments of the target data are adjusted in the storage node queue based on the number of access requests for the target data and the storage balance point.

5. The method according to claim 3, characterized in that, Based on the number of access requests for the target data and the storage balance point, the target storage nodes for multiple data segments of the target data are adjusted in the storage node queue, including: When the number of access requests for the target data is less than or equal to the storage balance point, any data segment of the target data will be stored sequentially in the storage nodes in the storage node queue. Specifically, when the remaining storage capacity of the current storage node in the storage node queue is less than or equal to the data capacity of the current data segment of the target data, the data to be migrated in the current storage node is determined according to the dynamic elastic window, and the data to be migrated is migrated to the storage node at the tail of the storage node queue until the remaining storage capacity of the current storage node is greater than the data capacity of the current data segment, and the current storage node is taken as the target storage node of the current data segment. When the number of access requests for the target data exceeds the storage balance point, the multi-cloud storage segmentation online algorithm strategy is re-executed for the target data.

6. The method according to claim 3, characterized in that, The implementation of allocation adjustment strategies also includes: Calculate the storage balance point; The formula for calculating the storage balance point is: Where Q is the target data; The data capacity of the j-th data segment of the target data; Let i be the unit price of storage for storage node i; The storage duration of the target data; The unit price for processing access requests; n is the number of storage nodes; m is the number of data segments of the target data.

7. The method according to claim 5, characterized in that, The implementation of allocation adjustment strategies also includes: Calculate the size of the dynamic elastic window, where the size of the dynamic elastic window represents the amount of data stored per unit time; The formula for calculating the size of the dynamic elastic window is: in, The size of the dynamic elastic window is T; T is the previous storage cycle. The number of access requests to the target data; The data capacity of the j-th data segment of the target data; Cost per access request processing; This is the sensitivity coefficient.

8. A multi-cloud storage processing device, characterized in that, The device includes: The acquisition module is used to acquire the target data to be stored and divide the target data into multiple data segments; The storage module is used to execute the multi-cloud storage segmentation online algorithm strategy to determine the target storage nodes for each of the multiple data segments corresponding to the target data, including executing the allocation strategy and executing the allocation adjustment strategy; The allocation strategy is used to determine the target storage nodes for each of the multiple data segments corresponding to the target data; the allocation adjustment strategy is used to adjust the target storage nodes of the multiple data segments of the target data according to the storage balance point when any access request exists, and the storage balance point is used to characterize the balance relationship between the number of access requests and the storage cost of the target data.

9. A computer device, characterized in that, include: At least one processor and a memory storing a computer program executable on the processor, wherein the processor executes the computer program to perform the multi-cloud storage processing method according to any one of claims 1-7.

10. A non-volatile readable storage medium, characterized in that, The non-volatile readable storage medium stores a computer program, wherein when the computer program is executed by a processor, it performs the multi-cloud storage processing method according to any one of claims 1-7.

Images