Data scheduling method, data access method, cloud storage platform and computing equipment

Abstract

The embodiment of the invention provides a data scheduling method, a data access method, a cloud storage platform and computing device.The data scheduling method is applied to a data scheduling end of the cloud storage platform and comprises the steps of obtaining historical use data of resources on at least one cloud storage service cluster, acquiring historical access data of a plurality of storage containers on at least one cloud storage service cluster; performing data analysis based on the historical use data and the historical access data, and predicting a matching relationship between the at least one cloud storage service cluster and the plurality of storage containers in a target time period; based on the matching relationship, the plurality of storage containers are deployed on the corresponding target cloud storage service clusters, and the resource use condition of each cloud storage service cluster and the access demand of the storage container are predicted in advance through data analysis of the historical use data and the historical access data, so that reasonable container deployment is completed in advance, and the user experience is improved. The problem of service response delay is solved, and the resource use efficiency and the overall performance of the cloud storage platform are improved.

Description

Technical Field

[0001] The embodiments in this specification relate to the technical field of cloud networks, and in particular to a data scheduling, data access method, cloud storage platform, and computing device. Background Technology

[0002] With the rapid development of cloud network technology and the massive migration of data to the cloud, cloud storage platforms, while providing efficient data storage and access services, also face enormous challenges, especially in terms of service quality and the rational utilization of resources.

[0003] Currently, traffic monitoring and alerts are generated by collecting real-time traffic data from multiple cloud storage service clusters in the cloud storage platform and comparing it with pre-set traffic thresholds. Based on this, data analysis is performed to identify the traffic sources of multiple cloud storage service clusters and analyze the available resources on multiple cloud storage service clusters, thereby completing traffic scheduling between multiple cloud storage service clusters.

[0004] However, the aforementioned traffic scheduling scheme only performs data analysis to complete temporary scheduling when a pre-set traffic threshold is triggered in real time. This results in a lag and is prone to performance bottlenecks due to insufficient available resources, causing service response delays and a decline in user experience. Furthermore, the frequent traffic scheduling across cloud storage service clusters increases the amount of additional scheduling traffic that consumes limited bandwidth resources, reducing the resource utilization efficiency of the cloud storage platform. Summary of the Invention

[0005] In view of this, embodiments of this specification provide a data scheduling method. One or more embodiments of this specification also relate to a data access method, a cloud storage platform, a data scheduling device, a data access device, a computing device, a computer-readable storage medium, and a computer program product, to address the technical deficiencies existing in the prior art.

[0006] According to a first aspect of the embodiments of this specification, a data scheduling method is provided, applied to a data scheduling terminal of a cloud storage platform, comprising:

[0007] Obtain historical usage data of resources on at least one cloud storage service cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster;

[0008] Based on historical usage and access data analysis, predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period.

[0009] Based on the matching relationship, multiple storage containers are deployed on the corresponding target cloud storage service cluster.

[0010] According to a second aspect of the embodiments of this specification, a data access method is provided, applied to a target cloud storage service cluster of a cloud storage platform, wherein the target cloud storage service cluster deploys corresponding target storage containers, and the matching relationship between the target storage containers and the target cloud storage service cluster is predicted based on data analysis of historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on at least one cloud storage service cluster. The method includes:

[0011] In response to data access requests sent by the terminal within the target time period, data access is performed on the object data in the target storage container.

[0012] According to a third aspect of the embodiments of this specification, a cloud storage platform is provided, including a data scheduling terminal and at least one cloud storage service cluster.

[0013] The data scheduling terminal is used to obtain historical usage data of resources on at least one cloud storage service cluster, as well as historical access data of multiple storage containers on at least one cloud storage service cluster. Based on the historical usage data and historical access data, data analysis is performed to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. Based on the matching relationship, multiple storage containers are deployed on the corresponding target cloud storage service cluster.

[0014] At least one cloud storage service cluster is used to access object data in deployed storage containers in response to data access requests sent by terminals within a target time period.

[0015] According to a fourth aspect of the embodiments of this specification, a data scheduling apparatus is provided, applied to the data scheduling terminal of a cloud storage platform, comprising:

[0016] The acquisition module is configured to acquire historical usage data of resources on at least one cloud storage service cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster.

[0017] The analysis module is configured to perform data analysis based on historical usage data and historical access data to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period.

[0018] The scheduling module is configured to deploy multiple storage containers on the corresponding target cloud storage service cluster based on matching relationships.

[0019] According to a fifth aspect of the embodiments of this specification, a data access apparatus is provided, applied to a target cloud storage service cluster of a cloud storage platform, wherein the target cloud storage service cluster deploys corresponding target storage containers, and the matching relationship between the target storage containers and the target cloud storage service cluster is predicted based on data analysis of historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on at least one cloud storage service cluster. The apparatus includes:

[0020] The access module is configured to access object data in the target storage container in response to data access requests sent by the terminal within the target time period.

[0021] According to a sixth aspect of the embodiments of this specification, a computing device is provided, comprising:

[0022] Memory and processor;

[0023] The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the above method.

[0024] According to a seventh aspect of the embodiments of this specification, a computer-readable storage medium is provided that stores a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0025] According to an eighth aspect of the embodiments of this specification, a computer program product is provided, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described method.

[0026] In one embodiment of this specification, a data scheduling method is provided, applied to the data scheduling terminal of a cloud storage platform, comprising: acquiring historical usage data of resources on at least one cloud storage service cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster; performing data analysis based on the historical usage data and historical access data to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period; and deploying multiple storage containers on the corresponding target cloud storage service cluster based on the matching relationship.

[0027] By analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period can be predicted in advance. This allows for the early determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and degraded user experience. At the same time, by pre-deploying storage containers on the corresponding target cloud storage service clusters, data within the same storage container can be accessed completely without cross-cluster access, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform. Attached Figure Description

[0028] Figure 1 This is a schematic diagram of the structure of a cloud storage platform;

[0029] Figure 2 This is a flowchart illustrating a data scheduling method provided in one embodiment of this specification;

[0030] Figure 3 This is a schematic diagram of the structure of a cloud storage platform in a data scheduling method provided in one embodiment of this specification;

[0031] Figure 4 This is a flowchart illustrating a data scheduling method provided in one embodiment of this specification;

[0032] Figure 5 This is a flowchart illustrating a data access method provided in one embodiment of this specification;

[0033] Figure 6 This is a flowchart illustrating the processing procedure of a data scheduling method for intelligent scheduling of traffic within the same city, provided in one embodiment of this specification.

[0034] Figure 7 This is a schematic diagram of the structure of a cloud storage platform provided in one embodiment of this specification;

[0035] Figure 8 This is a schematic diagram of the structure of a data scheduling device provided in one embodiment of this specification;

[0036] Figure 9 This is a schematic diagram of the structure of a data access device provided in one embodiment of this specification;

[0037] Figure 10 This is a structural block diagram of a computing device provided in one embodiment of this specification. Detailed Implementation

[0038] Many specific details are set forth in the following description to provide a full understanding of this specification. However, this specification can be implemented in many other ways than those described herein, and those skilled in the art can make similar extensions without departing from the spirit of this specification. Therefore, this specification is not limited to the specific implementations disclosed below.

[0039] The terminology used in one or more embodiments of this specification is for the purpose of describing particular embodiments only and is not intended to be limiting of the one or more embodiments of this specification. The singular forms “a,” “described,” and “the” as used in one or more embodiments of this specification and the appended claims are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in one or more embodiments of this specification refers to and includes any or all possible combinations of one or more associated listed items.

[0040] It should be understood that although the terms first, second, etc., may be used to describe various information in one or more embodiments of this specification, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from one another. For example, first may also be referred to as second without departing from the scope of one or more embodiments of this specification, and similarly, second may also be referred to as first. Depending on the context, the word "if" as used herein may be interpreted as "when," "when," or "in response to a determination."

[0041] Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in one or more embodiments of this specification are all information and data authorized by the user or fully authorized by all parties. Moreover, the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0042] First, the terms and concepts used in one or more embodiments of this specification will be explained.

[0043] Region: The region where multiple cloud storage service clusters are located.

[0044] Bucket: A data management unit for object data on a cloud storage platform accessed by users.

[0045] Internet Data Center (IDC): An Internet Data Center is a centralized location housing a large number of devices with computing and storage capabilities. It is typically used to provide data storage, backup, management, and application services. Data centers are a crucial component of cloud computing infrastructure, providing efficient and reliable data processing and storage services to businesses and individual users.

[0046] Availability Zone (AZ): A AZ is an independent area within a data center, formed through physical or logical isolation. Each AZ has its own independent power supply, cooling, and network facilities, designed to improve system availability and fault tolerance. Data synchronization and disaster recovery can be achieved between different AZs, ensuring service continuity and stability.

[0047] Grouping: A collection of multiple machines providing the same service and working collaboratively. In cloud storage platforms, front-end resources are organized into groups, providing services to the outside world based on task requirements. Machines within a group typically have similar configurations and functions, and they share service requests through load balancing technology, improving system response speed and reliability.

[0048] A cluster is a managed group of multiple machines that work together to provide a specific service. In cloud storage platforms, machines within a cluster share storage and computing resources, achieving high availability and high performance of data through distributed technologies. A cluster can contain multiple groups, each responsible for different services or tasks.

[0049] Front-end machine group: In a cloud storage platform, this is a group of machines responsible for receiving and processing user requests. These machines are located at the front end of the system and distribute requests through a load balancer to ensure fast response and high concurrency processing capabilities.

[0050] Backend cluster: In a cloud storage platform, a group of machines is responsible for data storage and processing. A backend cluster typically contains a large number of storage nodes, achieving efficient data storage and retrieval through a distributed file system or database management system. The design of a backend cluster needs to consider data security, consistency, and scalability.

[0051] Local network traffic: Data transmission traffic between different data centers or server rooms within the same city. Local network traffic typically has lower latency and higher bandwidth, making it suitable for scenarios requiring rapid data exchange and synchronization. By optimizing the management and scheduling of local network traffic, the overall performance and user experience of the cloud storage platform can be improved.

[0052] Currently, with the rapid development of cloud storage services, a large amount of user data is being migrated to the cloud. While cloud storage platforms provide efficient data storage and access services, they also face significant challenges, particularly in terms of service quality and efficient resource utilization. Especially in regions with high cluster density, users have extremely high requirements for data access performance, which a single cloud storage service cluster often struggles to meet. Therefore, building a cloud storage architecture with multiple cloud storage service clusters within the same city and implementing intelligent traffic scheduling solutions within the same region has become crucial for improving service quality.

[0053] Figure 1 A schematic diagram of the structure of a cloud storage platform is shown, such as... Figure 1 As shown:

[0054] For data access on cloud storage platforms, virtual IPs are allocated through the Domain Name System (DNS) to access object data stored in containers on multiple cloud storage service clusters.

[0055] Traffic monitoring and alerts for multiple cloud storage service clusters:

[0056] Real-time traffic data from multiple cloud storage service clusters in the cloud storage platform is collected and compared with pre-set traffic thresholds for traffic monitoring and alerts. Based on this, data analysis is performed to identify the traffic sources and available resources on the multiple cloud storage service clusters, enabling traffic scheduling between them. For example, if front-end group_1 accesses object data stored in a container in cloud storage back-end cluster_1, traffic scheduling will be performed to redirect the traffic from front-end group_1 to other clusters such as cloud storage back-end cluster_2 and cloud storage back-end cluster_3, thus completing cross-cluster traffic scheduling.

[0057] However, the aforementioned traffic scheduling scheme only performs analysis and temporary scheduling when a pre-set traffic threshold is triggered in real time. This results in a lag and is prone to performance bottlenecks due to insufficient available resources, causing service response delays and a decline in user experience. Furthermore, frequent cross-cloud storage service cluster traffic scheduling increases the amount of additional scheduling traffic used to occupy limited bandwidth resources, reducing the resource utilization efficiency of the cloud storage platform.

[0058] To address the aforementioned issues, this specification provides a data scheduling method, a data access method, a cloud storage platform, a data scheduling device, a data access device, a computing device, a computer-readable storage medium, and a computer program product, which will be described in detail in the following embodiments.

[0059] See Figure 2, Figure 2 This specification illustrates a flowchart of a data scheduling method according to an embodiment, applied to the data scheduling terminal of a cloud storage platform, including the following specific steps:

[0060] Step 202: Obtain historical usage data of resources on at least one cloud storage service cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster.

[0061] A cloud storage platform is a cloud computing infrastructure that provides data storage and access services. It stores data on remote servers via the internet, allowing users to access the data over the network. A cloud storage platform typically consists of multiple data centers, each containing multiple server rooms, and each server room contains multiple cloud storage service clusters. Through distributed storage technology and data scheduling mechanisms, cloud storage platforms provide users with large-scale, highly reliable, and high-performance data storage and access services. It supports various data types and access protocols to meet the needs of different task scenarios. Examples include cloud storage platforms for object data, block storage platforms, and file storage platforms.

[0062] The data scheduling module is a component in the cloud storage platform responsible for managing and scheduling data storage and access requests. It analyzes the resource usage and access demands of multiple cloud storage service clusters to determine appropriate data storage and access strategies. The data scheduling module monitors the real-time traffic and resource usage of each cloud storage cluster and performs intelligent scheduling to optimize performance and resource utilization. Typically located at the front end of the cloud storage platform, the data scheduling module works with the Domain Name System (DNS) to distribute user access requests to appropriate cloud storage service clusters. For example, the data scheduling module analyzes the resource usage and data access demands of storage containers on cloud storage service clusters (cluster1 and cluster2) to determine a data scheduling strategy: migrating some storage containers from cluster1 to cluster2 to balance the load on both clusters.

[0063] A cloud storage service cluster is a managed group of multiple machines working together to provide cloud storage services. Any cloud storage service cluster includes a front-end machine group and a cloud storage back-end cluster, achieving efficient data storage and access through distributed technology. Cloud storage service clusters are typically distributed across different data centers or server rooms to improve system availability and fault tolerance. Machines within a cloud storage service cluster usually share storage and computing resources, using load balancing technology to share service requests. For example, in a cloud storage platform, there might be three cloud storage service clusters: cluster1, cluster2, and cluster3, each containing dozens to hundreds of storage nodes responsible for storing and processing user data.

[0064] Resources refer to the hardware and software resources used to provide storage and computing services on a cloud storage service cluster, including but not limited to: Central Processing Unit (CPU), memory, disk, network bandwidth, operating system, middleware, database management system, and applications. For example, on a cloud storage service cluster, resources include: CPU, memory, disk, network bandwidth, and database management system.

[0065] Historical resource usage data refers to records of resource usage on a cloud storage service cluster over a historical period, including but not limited to: Queries Per Second (QPS), disk utilization, central processing unit utilization, memory utilization, and network bandwidth. For example, this could be the QPS, disk utilization, central processing unit utilization, memory utilization, and network bandwidth of a cloud storage service cluster over the past two weeks.

[0066] Storage containers are the basic units used to manage data in cloud storage platforms. Users access and manage data through storage containers. Storage containers typically have a unique identifier, serving as the fundamental unit for data storage and access. As the basic granularity for data scheduling, storage containers avoid the need to cross clusters to access data within the same storage container. Storage containers include, but are not limited to: buckets, blocks, and files. For example, in object storage, users can create multiple buckets, each storing multiple objects. In block storage, a storage container can be a volume, within which users can create and manage cloud disks. In file storage, a storage container can be a file system, within which users can create and manage files.

[0067] Historical access data for storage containers refers to records of data access behavior of storage containers on a cloud storage service cluster over a historical period, including but not limited to: access frequency, access time, access volume, and access source. For example, the access frequency, access time, access volume, and access source of a storage bucket on a cloud storage service cluster over the past two weeks.

[0068] It should be noted that the historical time periods for historical usage data and historical access data can be the same or different, and there are no restrictions.

[0069] To obtain historical usage data of resources on at least one cloud storage service cluster, one option is to extract historical usage data of resources on at least one cloud storage service cluster from the monitoring system of the cloud storage platform. Another option is to parse historical usage data of resources on at least one cloud storage service cluster from log files. Yet another option is to query historical usage data of resources on at least one cloud storage service cluster from the database. No specific method is specified here.

[0070] For example, the cloud storage platform for object data includes a data scheduling terminal and three cloud storage service clusters: cluster1, cluster2, and cluster3. These clusters provide the following services: Cluster1 primarily offers high availability and high performance data storage services, suitable for tasks requiring frequent read / write operations and low latency. Cluster2 primarily offers large-capacity storage services, suitable for tasks requiring large amounts of data storage. Cluster3 primarily offers high security and compliance data storage services, suitable for tasks with high data security and compliance requirements. The data scheduling terminal uses a point-based monitoring method to extract historical usage data from the cloud storage platform's monitoring system for the past two weeks from the resources on the three cloud storage service clusters: read / write queries per second, CPU utilization, memory utilization, disk utilization, and network bandwidth; and historical access data for the storage buckets on the three cloud storage service clusters for the past two weeks: access frequency, access time, and access volume.

[0071] Historical usage data of resources on at least one cloud storage service cluster, as well as historical access data of multiple storage containers on at least one cloud storage service cluster, are obtained to provide data support for subsequent analysis to obtain predicted usage data and predicted access data.

[0072] Step 204: Based on historical usage data and historical access data, perform data analysis to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period.

[0073] The target time period is the predicted future timeframe for data access. This period can be minutes, hours, days, or weeks, depending on the data access needs and scheduling strategies. Choosing an appropriate target time period allows for better adaptation to data access demands and scheduling strategies, ensuring the overall performance of the cloud storage platform.

[0074] The matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period is a scheduling strategy for deploying storage containers on the cloud storage service cluster within the target time period. This strategy is used to pre-deploy storage containers, ensuring that each storage container can run on a suitable cloud storage service cluster within the target time period. This avoids resource bottlenecks and service response delays, improves user experience and overall performance, and avoids frequent traffic scheduling across clusters, effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0075] Based on historical usage and access data, data analysis is performed to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. One possible approach is to perform data analysis based on historical usage and access data to obtain predicted usage and access data within the target time period, and then determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period based on the predicted usage and access data. For example, machine learning algorithms or statistical analysis algorithms can be used. Another possible approach is to perform data analysis based on historical usage and access data to directly predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period. For example, machine learning algorithms or statistical analysis algorithms can be used. No limitation is imposed here.

[0076] For example, machine learning algorithms are used to analyze historical usage data (queries per second, CPU utilization, memory utilization, disk utilization, and network bandwidth) of resources on three cloud storage service clusters over the past two weeks to directly predict the matching relationship between the three cloud storage service clusters and four storage containers within a target time period:

[0077] (bucket1, cloud storage service cluster2): Since the peak access times for bucket1 are 10:00-12:00 and 18:00-20:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket1 is deployed on cloud storage service cluster2, because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0078] (bucket2, cloud storage service cluster3): Since the peak access times for bucket2 are 12:00-14:00 and 20:00-22:00, and the resource utilization of cloud storage service cluster2 is high during these two time periods, bucket2 is deployed on cloud storage service cluster3, because the resource utilization of cloud storage service cluster3 is relatively low during these two time periods.

[0079] (Storage bucket 3, cloud storage service cluster 1): Since the peak access times for storage bucket 3 are 14:00-16:00 and 22:00-24:00, and the resource utilization of cloud storage service cluster 3 is relatively high during these two time periods, storage bucket 3 is deployed on cloud storage service cluster 1, because the resource utilization of cloud storage service cluster 1 is relatively low during these two time periods.

[0080] (bucket4, cloud storage service cluster2): Since the peak access times for bucket4 are 16:00-18:00 and 24:00-2:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket4 is deployed on cloud storage service cluster2 because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0081] Based on predicted usage data and predicted access data, the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period is determined, providing scheduling strategy support for the subsequent pre-deployment of storage containers.

[0082] Step 206: Based on the matching relationship, deploy multiple storage containers on the corresponding target cloud storage service cluster.

[0083] The target cloud storage service cluster is a cloud storage service cluster that has a matching relationship with storage containers. The target cloud storage service cluster is a cloud storage service cluster that is suitable for deploying storage containers within the target time period, determined based on the matching relationship.

[0084] Based on the matching relationship, multiple storage containers can be deployed on the corresponding target cloud storage service cluster. One possible approach is to migrate the data of multiple storage containers to the corresponding target cloud storage service cluster based on the matching relationship, and update the configuration of the domain name resolution system.

[0085] For example, for bucket1: current deployment location: cloud storage service cluster 1; target deployment location: cloud storage service cluster 2; migrate bucket1 from cloud storage service cluster 1 to cloud storage service cluster 2. After the migration is complete, update the domain name resolution system configuration to ensure that user access requests can be correctly routed to cloud storage service cluster 2.

[0086] For bucket2: Current deployment location: Cloud storage service cluster 2; Target deployment location: Cloud storage service cluster 3; Migrate bucket2 from Cloud storage service cluster 2 to Cloud storage service cluster 3. After the migration is complete, update the DNS resolution system configuration to ensure that user access requests can be correctly routed to Cloud storage service cluster 3.

[0087] For bucket3: Current deployment location: Cloud storage service cluster 3; Target deployment location: Cloud storage service cluster 1; Migrate bucket3 from Cloud storage service cluster 3 to Cloud storage service cluster 1. After the migration is complete, update the DNS resolution system configuration to ensure that user access requests can be correctly routed to Cloud storage service cluster 1.

[0088] For bucket 4: Current deployment location: Cloud storage service cluster 1; Target deployment location: Cloud storage service cluster 2; Migrate bucket 4 from Cloud storage service cluster 1 to Cloud storage service cluster 2. After the migration is complete, update the DNS resolution system configuration to ensure that user access requests can be correctly routed to Cloud storage service cluster 2.

[0089] In the embodiments described in this specification, by analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period are predicted in advance. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and decreased user experience. At the same time, by pre-deploying storage containers reasonably on the corresponding target cloud storage service clusters, data in the same storage container can be accessed completely without cross-cluster access, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0090] In one optional embodiment of this specification, step 204 includes the following specific steps:

[0091] Data analysis is performed on historical usage data to obtain predicted usage data of resources on each cloud storage service cluster within the target time period, and data analysis is performed on historical access data to obtain predicted access data of each storage container within the target time period.

[0092] Simulations are performed based on predicted usage data and predicted access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period.

[0093] Predicted resource usage data refers to forecasts of resource usage on a cloud storage service cluster within a target time period, including but not limited to: read / write queries per second, disk utilization, central processing unit utilization, memory utilization, and network bandwidth. This predicted resource usage data characterizes the resource capacity of the cloud storage service cluster. For example, it could be the predicted read / write queries per second, disk utilization, central processing unit utilization, memory utilization, and network bandwidth for a cloud storage service cluster over the next two weeks.

[0094] Predicted access data for storage containers refers to the predicted data access behavior of storage containers on a cloud storage service cluster within a target time period. This includes, but is not limited to, access frequency, access time, access volume, and access source. Predicted access data characterizes the future access behavior of storage containers. For example, it could represent the access frequency, access time, access volume, and access source of a storage bucket on a cloud storage service cluster over the next two weeks.

[0095] Data analysis of historical usage data can be used to obtain predicted usage data for resources on each cloud storage service cluster within the target time period. One possible approach is to perform time-series analysis on historical usage data to obtain predicted usage data for resources on each cloud storage service cluster within the target time period. Another possible approach is to perform machine learning analysis on historical usage data to obtain predicted usage data for resources on each cloud storage service cluster within the target time period. Yet another possible approach is to perform statistical analysis on historical usage data to obtain predicted usage data for resources on each cloud storage service cluster within the target time period. No specific approach is specified here.

[0096] Data analysis is performed on historical access data to obtain predicted access data for each storage container within the target time period. One possible approach is to perform time-series analysis on historical access data to obtain predicted access data for each storage container within the target time period. Another possible approach is to perform machine learning analysis on historical access data to obtain predicted access data for each storage container within the target time period. Yet another possible approach is to perform statistical analysis on historical access data to obtain predicted access data for each storage container within the target time period. No specific approach is specified here.

[0097] Simulations are performed based on predicted usage and access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. One possible approach is to perform simulations based on predicted usage and access data according to preset constraints. For example, the CPU utilization of the cloud storage service cluster is expected to not exceed 70% within the next two weeks, and simulations are performed based on this preset constraint. Another possible approach is to use a linear programming algorithm to perform simulations based on predicted usage and access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. For example, a cost function for resource utilization efficiency is defined, and simulations are performed by solving this function. Yet another possible approach is to use machine learning algorithms to perform simulations based on predicted usage and access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. For example, machine learning algorithms such as decision trees, random forests, support vector machines, or neural networks are used for simulations, and no limitation is imposed here.

[0098] For example, machine learning analysis is performed on the historical usage data (queries per second, CPU utilization, memory utilization, disk utilization, and network bandwidth) of resources on three cloud storage service clusters over the past two weeks to obtain predicted usage data for resources on the three cloud storage service clusters for the next two weeks:

[0099] Cloud storage service cluster 1: It is expected that the read / write QPS will increase from the current 1,000 times / second to 1,500 times / second in the next two weeks, the CPU utilization will increase from 40% to 60%, and the memory utilization will increase from 50% to 70%.

[0100] Cloud storage service cluster 2: It is expected that the read / write QPS will increase from the current 500 times / second to 800 times / second in the next two weeks, the CPU utilization will increase from 30% to 50%, and the memory utilization will increase from 40% to 60%.

[0101] Cloud storage service cluster 3: It is expected that the read / write QPS will increase from the current 300 times / second to 500 times / second in the next two weeks, the CPU utilization will increase from 20% to 40%, and the memory utilization will increase from 30% to 50%.

[0102] Machine learning analysis was performed on the historical access data (access frequency, access time, access volume) of storage buckets on three cloud storage service clusters over the past two weeks to obtain predicted access data for the storage buckets on the three cloud storage service clusters for the next two weeks:

[0103] Bucket 1: The access frequency is expected to increase from the current 100 times per hour to 150 times per hour in the next two weeks, with peak access times between 10:00-12:00 and 18:00-20:00.

[0104] Bucket2: The access frequency is expected to increase from the current 50 times per hour to 80 times per hour in the next two weeks, with peak access times between 12:00-14:00 and 20:00-22:00.

[0105] Bucket3: The access frequency is expected to increase from the current 30 times per hour to 50 times per hour in the next two weeks, with peak access times between 14:00-16:00 and 22:00-24:00.

[0106] Bucket4: The access frequency is expected to increase from the current 20 times per hour to 40 times per hour in the next two weeks, with peak access times between 16:00-18:00 and 24:00-2:00.

[0107] Define a cost function for resource utilization efficiency based on predicted usage data and predicted access data:

[0108]

[0109] Where N is the number of storage containers, M is the number of cloud storage service clusters, and f ij This indicates whether the i-th storage container is deployed on the j-th cloud storage service cluster (1 if deployed, 0 otherwise).

[0110] U i This represents the predicted average resource utilization of the j-th cloud storage service cluster within the target time period.

[0111] T j The threshold representing the ideal resource utilization rate of the j-th cloud storage service cluster.

[0112] P ij The penalty factor characterizes the deployment of the i-th storage container to the j-th cloud storage service cluster, which can be calculated based on the degree of mismatch between the storage container and the cluster in historical data.

[0113] ω1 and ω2 are weighting coefficients used to adjust the importance of different parts.

[0114] The purpose of this cost function is to minimize C, thereby optimizing resource utilization efficiency by rationally allocating storage containers.

[0115] By solving this function and conducting simulations, the matching relationship between three cloud storage service clusters and four storage containers within a target time period was determined:

[0116] (bucket1, cloud storage service cluster2): Since the peak access times for bucket1 are 10:00-12:00 and 18:00-20:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket1 is deployed on cloud storage service cluster2, because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0117] (bucket2, cloud storage service cluster3): Since the peak access times for bucket2 are 12:00-14:00 and 20:00-22:00, and the resource utilization of cloud storage service cluster2 is high during these two time periods, bucket2 is deployed on cloud storage service cluster3, because the resource utilization of cloud storage service cluster3 is relatively low during these two time periods.

[0118] (Storage bucket 3, cloud storage service cluster 1): Since the peak access times for storage bucket 3 are 14:00-16:00 and 22:00-24:00, and the resource utilization of cloud storage service cluster 3 is relatively high during these two time periods, storage bucket 3 is deployed on cloud storage service cluster 1, because the resource utilization of cloud storage service cluster 1 is relatively low during these two time periods.

[0119] (bucket4, cloud storage service cluster2): Since the peak access times for bucket4 are 16:00-18:00 and 24:00-2:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket4 is deployed on cloud storage service cluster2 because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0120] In the embodiments of this specification, historical usage data is analyzed to obtain predicted usage data of resources on each cloud storage service cluster within a target time period, and historical access data is analyzed to obtain predicted access data of each storage container within the target time period. This provides predictive data support for subsequently determining the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period. Based on the predicted usage data and predicted access data, the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period is determined, providing scheduling strategy support for subsequently pre-deploying storage containers.

[0121] In one optional embodiment of this specification, data analysis is performed on historical usage data to obtain predicted usage data for resources on each cloud storage service cluster within a target time period, including the following specific steps:

[0122] By integrating historical usage data into time-series data, a resource usage profile of resources on each cloud storage service cluster can be obtained.

[0123] Based on the resource usage profiles of resources on each cloud storage service cluster, machine learning models are used to predict the predicted resource usage data of each cloud storage service cluster within a target time period.

[0124] Accordingly, data analysis is performed on historical access data to obtain predicted access data for each storage container within the target time period, including the following specific steps:

[0125] By integrating historical access data into time-series data, a container access profile for each storage container can be obtained.

[0126] Based on the container access profiles of each storage container, machine learning models are used to predict the access data of each storage container within a target time period.

[0127] A resource usage profile is a multi-dimensional time-series feature set describing the usage of resources on a cloud storage service cluster over a historical period. The resource usage profile comprehensively reflects the usage patterns and trends of resources in different time periods, providing a foundation for predicting future resource usage. A resource usage profile typically includes multiple dimensions, such as time-series data on queries per second (QPS), disk utilization, central processing unit utilization, memory utilization, and network bandwidth. By integrating time-series data, a resource usage profile can be generated, which can then be used for training and prediction of machine learning models.

[0128] A container access profile is a multi-dimensional set of time-series features describing the access behavior of a storage container over a historical period. This profile comprehensively reflects the access patterns and trends of storage containers, providing a foundation for predicting future access demands. A container access profile typically includes multiple metrics, such as access frequency time-series data, access time time-series data, access volume time-series data, and access source time-series data. By integrating time-series data, a container access profile can be generated, which can then be used for training and prediction of machine learning models.

[0129] In cloud storage platforms, machine learning models can be used to predict resource usage and storage container access behavior, helping data schedulers make more accurate scheduling decisions. Machine learning models typically include various types such as supervised learning, unsupervised learning, and reinforcement learning. In cloud storage platforms, machine learning models can be linear regression models, decision tree models, random forest models, support vector machines (SVMs), neural network models, and so on. By training machine learning models, predictive models with predictive capabilities can be generated to predict future resource usage and storage container access behavior.

[0130] For example, historical usage data is integrated into time-series data to obtain a resource usage profile of the cloud storage service cluster cluster1: Read / Write QPS time-series data: QPS increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 1500 times / second; CPU utilization time-series data: CPU utilization increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 60%; Memory utilization... Time-series data on memory utilization: Memory utilization increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 70%; Time-series data on disk utilization: Disk utilization increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 50%; Time-series data on network bandwidth: Network bandwidth increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 200MB / s.

[0131] Similarly, resource usage profiles for the three cloud storage service clusters are obtained.

[0132] Based on resource usage profiles of three cloud storage service clusters, a neural network model is used to predict resource usage data for the next two weeks on these three clusters.

[0133] Cloud storage service cluster 1: It is expected that the read / write QPS will increase from the current 1,000 times / second to 1,500 times / second in the next two weeks, the CPU utilization will increase from 40% to 60%, and the memory utilization will increase from 50% to 70%.

[0134] Cloud storage service cluster 2: It is expected that the read / write QPS will increase from the current 500 times / second to 800 times / second in the next two weeks, the CPU utilization will increase from 30% to 50%, and the memory utilization will increase from 40% to 60%.

[0135] Cloud storage service cluster 3: It is expected that the read / write QPS will increase from the current 300 times / second to 500 times / second in the next two weeks, the CPU utilization will increase from 20% to 40%, and the memory utilization will increase from 30% to 50%.

[0136] Historical access data was integrated using time-series data to obtain a container access profile for bucket1: Access frequency time-series data: Access frequency increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 150 times / hour. Access time: Mainly concentrated between 10:00-12:00 and 18:00-20:00. Access volume time-series data: Access volume increases significantly during 10:00-12:00 and 18:00-20:00 daily, reaching a peak of 200GB.

[0137] Similarly, obtain the container access profiles for the four storage buckets.

[0138] Based on container access profiles of four storage buckets, a neural network model is used to predict the access data of the four storage buckets over the next two weeks.

[0139] Bucket 1: The access frequency is expected to increase from the current 100 times per hour to 150 times per hour in the next two weeks, with peak access times between 10:00-12:00 and 18:00-20:00.

[0140] Bucket2: The access frequency is expected to increase from the current 50 times per hour to 80 times per hour in the next two weeks, with peak access times between 12:00-14:00 and 20:00-22:00.

[0141] Bucket3: The access frequency is expected to increase from the current 30 times per hour to 50 times per hour in the next two weeks, with peak access times between 14:00-16:00 and 22:00-24:00.

[0142] Bucket4: The access frequency is expected to increase from the current 20 times per hour to 40 times per hour in the next two weeks, with peak access times between 16:00-18:00 and 24:00-2:00.

[0143] In the embodiments described in this specification, resource usage profiles and container access profiles are generated through time-series data integration and machine learning models, and future resource usage and storage container access behavior are predicted, providing more accurate predictive data support for subsequently determining the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period.

[0144] In one optional embodiment of this specification, simulation is performed based on predicted usage data and predicted access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period, including the following specific steps:

[0145] Under the premise of meeting the preset constraints of scheduling decision indicators, simulation is performed based on predicted usage data and predicted access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period. The scheduling decision indicators include at least one of cluster performance, resource utilization, resource availability and load balancing.

[0146] Scheduling decision metrics are at least one metric used to evaluate and select scheduling strategies. They help the data scheduler choose the appropriate strategy from multiple candidate options to optimize resource utilization and improve service quality. Scheduling decision metrics include at least one of cluster performance, resource utilization, resource availability, and load balancing.

[0147] Cluster performance refers to the performance of a cloud storage service cluster in handling data storage and access requests, including performance metrics such as response time, throughput, and concurrent processing capabilities. Cluster performance directly impacts user experience and task continuity. High cluster performance means lower response times and higher processing power, better meeting users' high-performance requirements for data access. For example, for a cloud storage service cluster, cluster performance might include: response time: average 10ms, maximum 20ms; throughput: 10,000 Input / Output Operations Per Second (IOPS); and concurrent processing capability: 1,000 concurrent connections.

[0148] Resource utilization is the ratio of actual resource usage to total resource capacity on a cloud storage service cluster. High resource utilization means resources are being fully utilized, reducing operating costs. However, excessively high resource utilization can lead to resource bottlenecks and impact performance. Therefore, reasonable resource utilization is crucial for ensuring system stability and performance. For example, for a cloud storage service cluster, resource utilization might include: CPU utilization: 50%; memory utilization: 60%; disk utilization: 40%; and network bandwidth utilization: 50%.

[0149] Resource availability refers to the degree to which resources on a cloud storage service cluster remain available and reliable within a target time period. High resource availability means that resources can continuously provide stable service without failures or unavailability during the target time period. Resource availability is a crucial indicator for ensuring task continuity and user experience. For example, for a cloud storage service cluster, resource availability includes: CPU availability: 99.5%; memory availability: 99.5%; disk availability: 99.5%; network bandwidth availability: 99.5%.

[0150] Load balancing distributes data storage and access requests reasonably across multiple cloud storage service clusters to prevent overload of any single cluster and ensure that the resource utilization and performance of each cluster remain at a reasonable operating level. Load balancing can improve the overall performance and reliability of the system, avoiding service interruptions or performance degradation caused by the overload of a single cluster. Load balancing can be achieved by dynamically adjusting the deployment location of storage containers. For example, for three cloud storage service clusters: Cluster 1: Current load: 80%, Maximum load: 100%; Cluster 2: Current load: 50%, Maximum load: 100%; Cluster 3: Current load: 60%, Maximum load: 100%.

[0151] For example, under the premise of meeting preset constraints on cluster performance, resource utilization, resource availability, and load balancing, simulation is performed based on predicted usage data and predicted access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period:

[0152] (bucket1, cloud storage service cluster2): Since the peak access times for bucket1 are 10:00-12:00 and 18:00-20:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket1 is deployed on cloud storage service cluster2, because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0153] (bucket2, cloud storage service cluster3): Since the peak access times for bucket2 are 12:00-14:00 and 20:00-22:00, and the resource utilization of cloud storage service cluster2 is high during these two time periods, bucket2 is deployed on cloud storage service cluster3, because the resource utilization of cloud storage service cluster3 is relatively low during these two time periods.

[0154] (Storage bucket 3, cloud storage service cluster 1): Since the peak access times for storage bucket 3 are 14:00-16:00 and 22:00-24:00, and the resource utilization of cloud storage service cluster 3 is relatively high during these two time periods, storage bucket 3 is deployed on cloud storage service cluster 1, because the resource utilization of cloud storage service cluster 1 is relatively low during these two time periods.

[0155] (bucket4, cloud storage service cluster2): Since the peak access times for bucket4 are 16:00-18:00 and 24:00-2:00, and the resource utilization of cloud storage service cluster1 is high during these two time periods, bucket4 is deployed on cloud storage service cluster2 because the resource utilization of cloud storage service cluster2 is relatively low during these two time periods.

[0156] In the embodiments described in this specification, under the condition of meeting the preset scheduling decision indicators, the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period is determined based on predicted usage data and predicted access data, providing a more accurate, comprehensive and reliable scheduling strategy for the subsequent pre-deployment of storage containers.

[0157] In one optional embodiment of this specification, the following specific steps are included before step 206:

[0158] Based on the matching relationship, determine whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period;

[0159] Correspondingly, step 206 includes the following specific steps:

[0160] If the conditions are met, multiple storage containers will be deployed on the corresponding target cloud storage service cluster based on the matching relationship.

[0161] For example, the matching relationship is as follows:

[0162] (bucket1, cloud storage service cluster2); (bucket2, cloud storage service cluster3); (bucket3, cloud storage service cluster1); (bucket4, cloud storage service cluster2).

[0163] Based on the matching relationship, determine whether the three cloud storage service clusters will meet the resource requirements of the corresponding storage containers within the next two weeks:

[0164] The cloud storage service cluster 2's read / write QPS, CPU utilization, and memory utilization during 10:00-12:00 and 18:00-20:00 all meet the access requirements of storage bucket 1.

[0165] The cloud storage service cluster 3's read / write QPS, CPU utilization, and memory utilization during 12:00-14:00 and 20:00-22:00 are sufficient to meet the access requirements of storage bucket 2.

[0166] The read / write QPS, CPU utilization, and memory utilization of the cloud storage service cluster 1 during 14:00-16:00 and 22:00-24:00 are within a reasonable range and can meet the access requirements of storage bucket 3.

[0167] The cloud storage service cluster cluster2's read / write QPS, CPU utilization, and memory utilization during 16:00-18:00 and 24:00-2:00 all meet the access requirements of storage bucket 4.

[0168] Based on the matching relationship, four storage containers are deployed on the corresponding target cloud storage service clusters.

[0169] In the embodiments of this specification, by determining whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period, and deploying the storage container on the target cloud storage service cluster if the conditions are met, the feasibility of the deployment is ensured, so that the deployed cloud storage service cluster can stably provide data access.

[0170] In one optional embodiment of this specification, after determining whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period based on the matching relationship, the following specific steps are further included:

[0171] If the requirements are not met, resource maintenance will be performed on the resources of the first target cloud storage service cluster, which is the target storage cloud service cluster that does not meet the resource requirements of the corresponding storage container.

[0172] One option for resource operation and maintenance on the first target cloud storage service cluster is to expand the resource capacity of the first target cloud storage service cluster. Another option is to optimize the resource capacity of the first target cloud storage service cluster, such as adjusting data access policies, optimizing data storage and access paths, and reducing unnecessary resource consumption. No specific method is specified here.

[0173] For example, based on matching relationships, it is determined whether three cloud storage service clusters will meet the resource requirements of their corresponding storage containers within the next two weeks:

[0174] During the periods of 12:00-14:00 and 20:00-22:00, the read / write CPU utilization and memory utilization of the cloud storage service cluster cluster3 cannot meet the access requirements of storage bucket2.

[0175] Expand the resources on the cloud storage service cluster cluster3: add 16 CPU cores and 64GB of memory.

[0176] In the embodiments of this specification, by determining whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period, and performing resource maintenance on the resources of the first target cloud storage service cluster if the conditions are not met, the feasibility of deployment is ensured, so that the deployed cloud storage service cluster can stably provide data access.

[0177] In one optional embodiment of this specification, the cloud storage platform further includes a resource pool;

[0178] Perform resource operation and maintenance on the resources of the first target cloud storage service cluster, including the following specific steps:

[0179] Determine if there are any idle resources in the resource pool within the target time period;

[0180] If so, based on available resources, expand the resources of the first target cloud storage service cluster.

[0181] A resource pool is a centrally managed collection of resources within a cloud storage platform that can be allocated to various cloud storage service clusters. These resources include, but are not limited to: Central Processing Units (CPUs), memory, disks, network bandwidth, operating systems, middleware, database management systems, and applications. The cloud storage platform can flexibly and dynamically allocate resources from the resource pool to cloud storage service clusters based on their needs, improving resource utilization and overall system performance. Through the resource pool, it can quickly respond to changes in resource demand, avoiding resource waste and shortages.

[0182] Idle resources are resources in the resource pool that are not currently allocated. Idle resources can be allocated to cloud storage service clusters that require them to cope with changes in resource demand. The existence of idle resources enables the cloud storage platform to scale up in a timely manner during peak resource demand periods, improving the stability and availability of the cloud storage platform.

[0183] For example, determine whether there are idle resources in the resource pool during the target time period: CPU: 64 cores used, 64 cores remaining; Memory: 256GB used, 256GB remaining; Disk space: 5TB used, 5TB remaining; Network bandwidth: 5Gbps used, 5Gbps remaining. Allocate 16 CPU cores and 64GB of memory from the resource pool to the cloud storage service cluster cluster3 to meet the access requirements of storage bucket2.

[0184] In the embodiments described in this specification, through centralized management and dynamic allocation of resource pools, the cloud storage platform can quickly respond to changes in resource demand, avoid resource waste and shortage, and improve the stability and availability of the cloud storage platform.

[0185] In one optional embodiment of this specification, after determining whether there are idle resources in the resource pool within the target time period, the following specific steps are further included:

[0186] If not, replenish the resource pool;

[0187] Based on the supplemented resource pool, the resources of the first target cloud storage service cluster are expanded.

[0188] To replenish resources in the resource pool, one option is to trigger a resource replenishment process to obtain resources from external resources. Another option is to generate a resource replenishment message and manually add resources. Yet another option is to allocate resources from other cloud storage service clusters to the resource pool. No specific method is specified here.

[0189] For example, if not, the resource replenishment process is automatically triggered to purchase a 16-core CPU and 64GB of memory from the cloud service provider and add them to the resource pool. Based on the replenished resource pool, the 16-core CPU and 64GB of memory are allocated from the resource pool to the cloud storage service cluster cluster3 to meet the access requirements of storage bucket2.

[0190] The embodiments in this specification effectively avoid resource bottlenecks and service response delays, thereby improving user experience and overall performance.

[0191] In one optional embodiment of this specification, after step 206, the following specific steps are further included:

[0192] If the deployment fails, update the historical usage data of the resources on the second target cloud storage service cluster, and return to the step of performing data analysis based on the historical usage data and historical access data to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period, wherein the second target cloud storage service cluster is the target storage cloud service cluster that failed to deploy.

[0193] Although attempts are made to deploy multiple storage containers on the corresponding target cloud storage service cluster based on the matching relationship, deployment may fail due to hardware or other reasons. In such cases, the historical usage data of the resources on the target cloud storage service cluster where the deployment failed will be updated, and the process will return to the step of determining the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period based on the predicted usage data and predicted access data, and the scheduling decision will be re-made.

[0194] For example, bucket 1 is successfully deployed on cloud storage service cluster 2. When attempting to deploy bucket 2 to cloud storage service cluster 3, the deployment fails due to disk corruption. Bucket 3 is successfully deployed on cloud storage service cluster 1. Bucket 4 is successfully deployed on cloud storage service cluster 2. Upon detecting the deployment failure of bucket 2 on cloud storage service cluster 3, the historical usage data of resources on cloud storage service cluster 3 is updated, and the process returns to the step of determining the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period based on predicted usage data and predicted access data. Bucket 2 is then deployed on cloud storage service cluster 1.

[0195] In the embodiments described in this specification, by returning to the execution of the matching relationship determination step based on predicted usage data and predicted access data when deployment fails, a new scheduling decision is made to obtain a reasonable scheduling strategy, which effectively avoids resource bottlenecks and service response delays, and improves user experience and overall performance.

[0196] In one optional embodiment of this specification, any cloud storage service cluster includes a front-end machine group and a cloud storage back-end cluster;

[0197] Accordingly, step 202 involves obtaining historical usage data of resources on at least one cloud storage service cluster, including the following specific steps:

[0198] Obtain historical usage data of resources on at least one front-end machine group and historical usage data of resources on at least one cloud storage back-end cluster;

[0199] Accordingly, step 204 includes the following specific steps:

[0200] Based on historical usage data of the front end and the back end, data analysis is performed to predict the matching relationship between at least one front end machine group and at least one cloud storage back end cluster, and multiple storage containers within a target time period.

[0201] Accordingly, step 206 includes the following specific steps:

[0202] Based on the matching relationship, multiple storage containers are deployed on the corresponding target front-end machine group and target cloud storage back-end cluster.

[0203] A front-end machine group is a set of machines in a cloud storage platform responsible for receiving and processing user requests. These machines are located at the front end of the system and distribute requests through a load balancer to ensure fast response and high concurrency processing capabilities. For example, in a cloud storage platform, front-end machine group_1 contains 10 front-end machines, each responsible for processing user requests and distributing them to the back-end cluster through a load balancer.

[0204] A cloud storage backend cluster is a group of machines within a cloud storage platform responsible for data storage and processing. A backend cluster typically contains a large number of storage nodes, achieving efficient data storage and retrieval through a distributed file system or database management system. The design of a backend cluster needs to consider data security, consistency, and scalability. For example, in a cloud storage platform, cloud storage backend cluster_1 contains 50 storage nodes, each responsible for storing and processing user data, and achieving efficient data storage and retrieval through a distributed file system.

[0205] Historical resource usage data refers to records of resource usage on front-end machine groups over a historical period. This historical usage data includes, but is not limited to, queries per second (QPS), disk utilization, central processing unit (CPU) utilization, memory utilization, and network bandwidth. For example, read / write QPS: Front-end machine group_1's average read / write QPS over the past two weeks is 1000 times / second; CPU utilization: Front-end machine group_1's average CPU utilization over the past two weeks is 40%; Memory utilization: Front-end machine group_1's average memory utilization over the past two weeks is 50%; Network bandwidth: Front-end machine group_1's average network bandwidth utilization over the past two weeks is 60%.

[0206] Historical resource usage data refers to records of resource usage on the cloud storage backend cluster over a historical period. This historical usage data includes, but is not limited to, queries per second (QPS), disk utilization, central processing unit (CPU) utilization, memory utilization, and network bandwidth. For example, read / write QPS: Cloud storage backend cluster_1's average read / write QPS over the past two weeks is 500 times / second; CPU utilization: Cloud storage backend cluster_1's average CPU utilization over the past two weeks is 30%; Memory utilization: Cloud storage backend cluster_1's average memory utilization over the past two weeks is 40%; Disk utilization: Cloud storage backend cluster_1's average disk utilization over the past two weeks is 70%; Network bandwidth: Cloud storage backend cluster_1's average network bandwidth utilization over the past two weeks is 50%.

[0207] The target front-end machine group is a front-end machine group that has a matching relationship with the storage container. The target front-end machine group is a front-end machine group that is suitable for deploying storage containers within the target time period, determined based on the matching relationship.

[0208] The target cloud storage backend cluster is a cloud storage backend cluster that has a matching relationship with storage containers. The target cloud storage backend cluster is a cloud storage backend cluster that is suitable for deploying storage containers within the target time period, determined based on the matching relationship.

[0209] For details not described in detail in the embodiments of this specification, please refer to the description of steps 202-206 above, and they will not be repeated here.

[0210] In the embodiments described in this specification, by analyzing historical usage data of the front-end, back-end, and access data, the resource usage and access requirements of storage containers in the front-end machine groups and cloud storage back-end clusters of each cloud storage service cluster are predicted in advance within a target time period. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of temporary scheduling and solving the problems of service response delay and user experience degradation. At the same time, by pre-deploying storage containers reasonably on the corresponding target front-end machine groups and target cloud storage back-end clusters, access to data in the same storage container can be completed without cross-cluster access, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0211] In one optional embodiment of this specification, data analysis is performed based on historical usage data of the front end and the back end to predict the matching relationship between at least one front-end machine group and at least one cloud storage back-end cluster and multiple storage containers within a target time period. This includes the following specific steps:

[0212] Data analysis is performed on historical front-end usage data to obtain the predicted front-end usage data of resources on each front-end machine group within the target time period. Data analysis is also performed on historical back-end usage data to obtain the predicted back-end usage data of resources on each cloud storage back-end cluster within the target time period.

[0213] Simulations are performed based on front-end predicted usage data, back-end predicted usage data, and predicted access data to determine the matching relationship between at least one front-end machine group, at least one cloud storage back-end cluster, and multiple storage containers within a target time period.

[0214] The front-end usage forecast data is the predicted usage data of resources on the front-end machine group within a target time period. This forecast data includes, but is not limited to, queries per second (QPS), disk utilization, central processing unit (CPU) utilization, memory utilization, and network bandwidth. The front-end usage forecast data characterizes the resource capabilities of the front-end machine group on the cloud storage service cluster. For example, read / write QPS: The predicted read / write QPS for front-end machine group_1 will increase from the current 1000 queries / second to 1500 queries / second in the next two weeks; CPU utilization: The predicted CPU utilization for front-end machine group_1 will increase from the current 40% to 60% in the next two weeks; memory utilization: The predicted memory utilization for front-end machine group_1 will increase from the current 50% to 70% in the next two weeks; network bandwidth: The predicted network bandwidth utilization for front-end machine group_1 will increase from the current 60% to 80% in the next two weeks.

[0215] The predicted backend usage data for resources refers to the projected usage of resources on the cloud storage backend cluster within a target time period. This predicted usage data includes, but is not limited to, queries per second (QPS), disk utilization, central processing unit (CPU) utilization, memory utilization, and network bandwidth. The predicted backend usage data characterizes the resource capabilities of the cloud storage backend cluster on the cloud storage service cluster. For example, read / write QPS: The predicted read / write QPS for cloud storage backend cluster_1 will increase from the current 500 queries / second to 800 queries / second in the next two weeks; CPU utilization: The predicted CPU utilization for cloud storage backend cluster_1 will increase from the current 30% to 50% in the next two weeks; memory utilization: The predicted memory utilization for cloud storage backend cluster_1 will increase from the current 40% to 60% in the next two weeks; disk utilization: The predicted disk utilization for cloud storage backend cluster_1 will increase from the current 70% to 80% in the next two weeks; network bandwidth: The predicted network bandwidth utilization for cloud storage backend cluster_1 will increase from the current 50% to 70% in the next two weeks.

[0216] For details not described in detail in the steps of the embodiments of this specification, please refer to the description of the above-mentioned response specification embodiments, and they will not be repeated here.

[0217] In the embodiments of this specification, data analysis is performed on historical usage data of the front-end and back-end to obtain predicted usage data of front-end and back-end resources on each cloud storage service cluster within a target time period. Data analysis is also performed on historical access data to obtain predicted access data of each storage container within the target time period. This provides predictive data support for subsequently determining the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period. Based on the predicted usage data of the front-end and back-end and the predicted access data, the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period is determined, providing scheduling strategy support for subsequently pre-deploying storage containers.

[0218] In one optional embodiment of this specification, at least one cloud storage service cluster is located in the same region.

[0219] For example, cloud storage service cluster 1: Data center 1 located in city A, mainly provides high availability and high performance data storage services, suitable for task scenarios that require frequent read and write and low latency.

[0220] Cloud storage service cluster 2: Located in City A, this data center primarily provides high-capacity storage services, suitable for scenarios requiring the storage of large amounts of data.

[0221] Cloud storage service cluster 3: Located in City A, data center 3 primarily provides high-security and compliant data storage services, suitable for task scenarios with high requirements for data security and compliance.

[0222] In the embodiments described in this specification, by scheduling data among cloud storage service clusters located in the same region, network latency is reduced, user experience is improved, the system's fault tolerance and availability are enhanced, traffic allocation is dynamically adjusted to ensure load balancing among clusters, resource bottlenecks are avoided, and data consistency and reliability can be achieved more efficiently.

[0223] against Figure 2 The cloud storage platform described in the embodiments of the specification Figure 3 This specification illustrates a schematic diagram of the structure of a cloud storage platform in a data scheduling method according to an embodiment of the present specification:

[0224] For data access on cloud storage platforms, virtual IPs are allocated through the domain name resolution system to access object data stored in containers on multiple cloud storage service clusters.

[0225] Traffic monitoring and alerts for multiple cloud storage service clusters:

[0226] On the data scheduling end, historical usage data of resources on at least one front-end machine group, historical usage data of resources on at least one cloud storage back-end cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster are collected.

[0227] The system integrates historical usage data from the front end to obtain a front end resource usage profile. Based on this profile, a machine learning model is used to predict the front end resource usage data for the front end machine group within a target time period. Similarly, it integrates historical usage data from the back end to obtain a back end resource usage profile. Based on this profile, a machine learning model is used to predict the back end resource usage data for the cloud storage back end cluster within a target time period. Finally, it integrates historical access data to obtain a container access profile for each storage container. Based on this profile, a machine learning model is used to predict the access data for each storage container within a target time period.

[0228] Based on front-end predicted usage data, back-end predicted usage data, and predicted access data, determine the matching relationship between at least one front-end machine group and at least one cloud storage back-end cluster, and multiple storage containers within a target time period.

[0229] Based on the matching relationship, it is determined whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period. If it does, multiple storage containers are deployed on the corresponding target cloud storage service cluster based on the matching relationship. If it does not, the target cloud storage service cluster is expanded based on the idle resources of the resource pool within the target time period.

[0230] For example, front-end group _1 only needs to access object data pre-stored in storage containers within the cloud storage back-end cluster _1, without requiring cross-cluster traffic scheduling. This achieves intelligent scheduling, reducing the costs associated with intra-city traffic while ensuring service quality.

[0231] against Figure 2 The execution flow of the embodiments in the specification, Figure 4 A flowchart illustrating a data scheduling method according to an embodiment of this specification is shown below:

[0232] Collect historical access data for storage containers 1 and 2, and collect historical usage data for resources on cloud storage service cluster 1 and cloud storage service cluster 2.

[0233] Based on historical access data, a container access profile is constructed for the storage container. The container access profile includes: access model, behavioral characteristics, and peak data.

[0234] Based on historical usage data, a resource usage profile is constructed, which includes: access QPS, disk utilization, central processing unit, memory, network bandwidth, etc.

[0235] Using machine learning models, based on container access profiles, predictive access data for storage containers is generated.

[0236] Using machine learning models, based on resource usage profiles, predictive usage data for resources is generated.

[0237] The data scheduling end uses a decision algorithm to complete data scheduling based on predicted access data and predicted usage data, and further completes the operation and maintenance of traffic within the same city.

[0238] See Figure 5 , Figure 5 The flowchart illustrates a data access method according to an embodiment of this specification, applied to a target cloud storage service cluster on a cloud storage platform. The target cloud storage service cluster deploys corresponding target storage containers. The matching relationship between the target storage containers and the target cloud storage service cluster is predicted based on data analysis of historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on at least one cloud storage service cluster. The method includes the following specific steps:

[0239] Step 502: In response to the data access request sent by the terminal within the target time period, access the object data in the target storage container.

[0240] A terminal is a device or application used by a user to access a cloud storage platform. Terminals can be various types of devices, including but not limited to personal computers, smartphones, tablets, and servers, as well as various applications such as web browsers, mobile applications, and enterprise software. Terminals access object data from the cloud storage platform by initiating data access requests.

[0241] A data access request is a request sent by a terminal to a cloud storage platform to access object data in a storage container. A data access request typically includes the request method (such as GET, POST, PUT, DELETE, etc.), the Uniform Resource Locator (URL) of the request, request headers (such as authentication information, content type, etc.), and the request body (such as the object data to be accessed). For example, a GET request: The terminal sends a GET request to retrieve object data from the storage bucket from the cloud storage platform.

[0242] GET / bucket1 / object1 HTTP / 1.1 / / **Data access path

[0243] Host: storage.example.com / / **The host domain name of the cloud storage service cluster

[0244] Authorization: Bearer<access_token> / / **Access permissions

[0245] POST request: The terminal sends a POST request to upload new object data to the cloud storage platform.

[0246] POST / bucket1 / object2 HTTP / 1.1 / / **Data access path

[0247] Host: storage.example.com / / **The host domain name of the cloud storage service cluster

[0248] Authorization: Bearer<access_token> / / **Access permissions

[0249] Content-Type: application / octet-stream / / ** Access data type: data stream

[0250] Content-Length: 1024 / / ** Access data length: 1024

[0251] [Binary Data] / / **Binary data

[0252] PUT Request: The terminal sends a PUT request to update existing object data in the cloud storage platform.

[0253] PUT / bucket1 / object3 HTTP / 1.1 / / **Data access path

[0254] Host: storage.example.com / / **The host domain name of the cloud storage service cluster

[0255] Authorization: Bearer<access_token> / / **Access permissions

[0256] Content-Type: application / json / / ** Access data type: json file

[0257] Content-Length: 512 / / ** Access data length: 512

[0258] {"key": "value"} / / **Key-value pair data

[0259] DELETE request: The terminal sends a DELETE request to delete object data from the cloud storage platform.

[0260] DELETE / bucket1 / object4 HTTP / 1.1 / / **Data access path

[0261] Host: storage.example.com / / **The host domain name of the cloud storage service cluster

[0262] Authorization: Bearer<access_token> / / **Access permissions

[0263] Object data refers to data stored in storage containers on a cloud storage platform. Object data can be various types of files, such as text files, images, videos, and documents. Each object typically has a unique identifier (such as the object name) and metadata (such as creation time, modification time, size, etc.). For example: a text file: object1.txt in bucket1, containing a piece of text data. an image file: image1.jpg in bucket2, containing an image. a video file: video1.mp4 in bucket3, containing a video clip. a JSON file: data1.json in bucket4, containing JSON formatted data.

[0264] It should be noted that the technical solution of this data access method and the technical solution of the aforementioned data scheduling method belong to the same concept. For details not described in detail in the technical solution of the data access method, please refer to the description of the technical solution of the aforementioned data scheduling method, which will not be repeated here.

[0265] In the embodiments described in this specification, by analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period are predicted in advance. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and decreased user experience. At the same time, by pre-deploying storage containers reasonably on the corresponding target cloud storage service clusters, terminals can access object data in the target storage containers completely without cross-cluster access within the target time period, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0266] The following is in conjunction with the appendix Figure 6 Taking the data scheduling method provided in this manual as an example of its application in intelligent traffic scheduling within the same city, the data scheduling method will be further explained. Figure 6 This specification illustrates a flowchart of a data scheduling method for intelligent traffic scheduling within the same city, provided in one embodiment. The method is applied to the data scheduling terminal of a cloud storage platform located within the same city and includes the following specific steps:

[0267] Step 602: Collect historical front-end usage data such as read / write QPS, disk utilization, CPU utilization, memory utilization, and network bandwidth of the front-end machine group over the past two weeks. Integrate the historical front-end usage data of the front-end machine group over the past two weeks into time series data to obtain the front-end resource usage profile of the front-end machine group. Based on the front-end resource usage profile, use a machine learning model to predict the front-end predicted resource usage data for the next two weeks.

[0268] Step 604: Collect historical backend usage data of the cloud storage backend cluster over the past two weeks, such as read / write QPS, disk utilization, CPU utilization, memory utilization, and network bandwidth. Integrate the historical backend usage data of the cloud storage backend cluster over the past two weeks into time series data to obtain a backend resource usage profile of the cloud storage backend cluster. Based on the backend resource usage profile, use a machine learning model to predict the backend resource usage data for the next two weeks.

[0269] Step 606: Collect historical access data of the storage bucket over the past two weeks, such as read / write model, access peak, and access behavior characteristics. Integrate the historical access data of the storage bucket over the past two weeks into time series data to obtain a container access profile of the storage bucket. Use a machine learning model to predict the predicted access data of the storage bucket for the next two weeks.

[0270] Step 608: Under the premise of meeting the preset constraints of the preset scheduling decision indicators, simulation is performed based on the predicted usage data of the front end, the predicted usage data of the back end, and the predicted access data to generate a scheduling strategy. The scheduling strategy includes the matching relationship between the front end machine group and the cloud storage back end cluster and the storage bucket in the next two weeks.

[0271] Step 610: Based on the scheduling strategy, migrate the storage bucket to the corresponding front-end machine and cloud storage back-end cluster;

[0272] Step 612: If it is found that there is same-city traffic in the data access for object data in the storage bucket, perform resource operation and maintenance: Determine whether the cloud storage service cluster can meet the resource requirements of the corresponding storage container in the next two weeks. If not, determine whether there are idle resources in the resource pool in the next two weeks. If yes, expand the resources. If no, trigger the resource replenishment process. If the resource cannot be expanded due to hardware reasons, return to step 608.

[0273] In the embodiments of this specification, a resource capability assessment model for the front-end and back-end of the cloud storage service cluster is established to calculate the service capabilities of the front-end and back-end of the cluster in the future time period. The resources of the front-end and back-end of the cluster are rationally utilized. By analyzing the historical access data of the storage buckets, a container access profile of the storage buckets is established. The access patterns and trends of the storage buckets are analyzed in depth. Furthermore, access hotspots in different time periods are explored in depth to optimize the layout of storage buckets on the same-city cloud storage service cluster. Machine learning is used to analyze and predict the resource capabilities and container access profiles of the front-end and back-end of the cluster. Scheduling decisions and resource operation and maintenance are formulated at the storage bucket level to ensure service quality while controlling the costs brought by same-city traffic.

[0274] Corresponding to the above method embodiments, this specification also provides cloud storage platform embodiments. Figure 7A schematic diagram of the structure of a cloud storage platform provided in one embodiment of this specification is shown. Figure 7 As shown, the cloud storage platform 700 includes a data scheduling terminal 702 and at least one cloud storage service cluster 704;

[0275] The data scheduling terminal 702 is used to obtain historical usage data of resources on at least one cloud storage service cluster 704, as well as historical access data of multiple storage containers on at least one cloud storage service cluster 704. It performs data analysis on the historical usage data to obtain predicted usage data of resources on each cloud storage service cluster 704 within a target time period, and performs data analysis on the historical access data to obtain predicted access data of each storage container within the target time period. Based on the predicted usage data and predicted access data, it determines the matching relationship between at least one cloud storage service cluster 704 and multiple storage containers within the target time period, and deploys multiple storage containers on the corresponding cloud storage service cluster 704 based on the matching relationship.

[0276] At least one cloud storage service cluster 704 is used to access object data in the deployed storage container in response to data access requests sent by the terminal within a target time period.

[0277] In the embodiments described in this specification, by analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period are predicted in advance. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and decreased user experience. At the same time, by pre-deploying storage containers reasonably on the corresponding target cloud storage service clusters, data in the same storage container can be accessed completely without cross-cluster access, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0278] The above is an illustrative scheme of a cloud storage platform according to this embodiment. It should be noted that the technical solution of this cloud storage platform belongs to the same concept as the technical solutions of the data scheduling method and data access method described above. For details not described in detail in the technical solution of the cloud storage platform, please refer to the description of the technical solutions of the data scheduling method or data access method described above.

[0279] Corresponding to the above method embodiments, this specification also provides embodiments of a data scheduling device. Figure 8 A schematic diagram of a data scheduling device according to one embodiment of this specification is shown. Figure 8 As shown, this device is used in the data scheduling terminal of a cloud storage platform and includes:

[0280] The acquisition module 802 is configured to acquire historical usage data of resources on at least one cloud storage service cluster, and historical access data of multiple storage containers on at least one cloud storage service cluster.

[0281] Analysis module 804 is configured to perform data analysis based on historical usage data and historical access data to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period.

[0282] The scheduling module 806 is configured to deploy multiple storage containers on the corresponding target cloud storage service cluster based on matching relationships.

[0283] Optionally, the analysis module 804 is further configured to: perform data analysis on historical usage data to obtain predicted usage data of resources on each cloud storage service cluster within the target time period, and perform data analysis on historical access data to obtain predicted access data of each storage container within the target time period; and perform simulation based on the predicted usage data and predicted access data to determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within the target time period.

[0284] Optionally, the analysis module 804 is further configured to: integrate historical usage data into time series data to obtain resource usage profiles of resources on each cloud storage service cluster; based on the resource usage profiles of resources on each cloud storage service cluster, use machine learning models to predict the predicted usage data of resources on each cloud storage service cluster within a target time period; integrate historical access data into time series data to obtain container access profiles of each storage container; based on the container access profiles of each storage container, use machine learning models to predict the predicted access data of each storage container within a target time period.

[0285] Optionally, the analysis module 804 is further configured to: perform simulation based on predicted usage data and predicted access data, under the condition of satisfying the preset constraints of the scheduling decision indicators, and determine the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period, wherein the scheduling decision indicators include at least one of cluster performance, resource utilization, resource availability and load balancing.

[0286] Optionally, the device further includes: an evaluation module configured to determine, based on the matching relationship, whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within a target time period;

[0287] Correspondingly, the scheduling module 806 is further configured to: if the conditions are met, deploy multiple storage containers on the corresponding target cloud storage service cluster based on the matching relationship.

[0288] Optionally, the device further includes an operation and maintenance module configured to perform resource operation and maintenance on the resources on the first target cloud storage service cluster if the requirements are not met, wherein the first target cloud storage service cluster is a target storage cloud service cluster that does not meet the resource requirements of the corresponding storage container.

[0289] Optionally, the cloud storage platform also includes a resource pool; correspondingly, the operation and maintenance module is further configured to: determine whether there are idle resources in the resource pool during the target time period; if so, expand the resources of the first target cloud storage service cluster based on the idle resources.

[0290] Optionally, the device further includes: a supplementation module configured to supplement the resource pool if not, and to expand the resources of the first target cloud storage service cluster based on the supplemented resource pool.

[0291] Optionally, the apparatus further includes a re-decision module configured to, if deployment fails, update the historical usage data of resources on the second target cloud storage service cluster, and return to the step of performing data analysis based on historical usage data and historical access data to predict the matching relationship between at least one cloud storage service cluster and multiple storage containers within a target time period, wherein the second target cloud storage service cluster is the target storage cloud service cluster whose deployment failed.

[0292] Optionally, any cloud storage service cluster includes a front-end group and a cloud storage back-end cluster;

[0293] Correspondingly, the acquisition module 802 is further configured to: acquire the front-end historical usage data of resources on at least one front-end machine group, and the back-end historical usage data of resources on at least one cloud storage back-end cluster;

[0294] Correspondingly, the analysis module 804 is further configured to: perform data analysis based on the historical usage data of the front end and the historical usage data of the back end, and predict the matching relationship between at least one front end machine group and at least one cloud storage back end cluster and multiple storage containers within the target time period;

[0295] Correspondingly, the scheduling module 806 is further configured to deploy multiple storage containers on the corresponding target front-end machine group and target cloud storage back-end cluster based on the matching relationship.

[0296] In the embodiments described in this specification, by analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period are predicted in advance. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and decreased user experience. At the same time, by pre-deploying storage containers reasonably on the corresponding target cloud storage service clusters, data in the same storage container can be accessed completely without cross-cluster access, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0297] The above is an illustrative scheme of a data scheduling device according to this embodiment. It should be noted that the technical solution of this data scheduling device and the technical solution of the data scheduling method described above belong to the same concept. For details not described in detail in the technical solution of the data scheduling device, please refer to the description of the technical solution of the data scheduling method described above.

[0298] Corresponding to the above method embodiments, this specification also provides data access device embodiments. Figure 9 A schematic diagram of a data access device according to one embodiment of this specification is shown. Figure 9 As shown, the device is applied to a target cloud storage service cluster on a cloud storage platform. The target cloud storage service cluster deploys corresponding target storage containers. The matching relationship between the target storage containers and the target cloud storage service cluster is predicted based on data analysis of historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on at least one cloud storage service cluster. The device includes:

[0299] Access module 902 is configured to access object data in the target storage container in response to a data access request sent by the terminal within the target time period.

[0300] In the embodiments described in this specification, by analyzing historical usage and access data, the resource usage of each cloud storage service cluster and the access demand of storage containers within a target time period are predicted in advance. This allows for the pre-determination of reasonable matching relationships to complete the deployment of storage containers, avoiding the lag of ad-hoc scheduling and solving the problems of service response delays and decreased user experience. At the same time, by pre-deploying storage containers reasonably on the corresponding target cloud storage service clusters, terminals can access object data in the target storage containers completely without cross-cluster access within the target time period, reducing frequent cross-cluster traffic scheduling and effectively improving the resource utilization efficiency and overall performance of the cloud storage platform.

[0301] The above is an illustrative scheme of a data access device according to this embodiment. It should be noted that the technical solution of this data access device and the technical solution of the data access method described above belong to the same concept. For details not described in detail in the technical solution of the data access device, please refer to the description of the technical solution of the data access method described above.

[0302] Figure 10 This specification illustrates a structural block diagram of a computing device according to one embodiment. The components of the computing device 1000 include, but are not limited to, a memory 1010 and a processor 1020. The processor 1020 is connected to the memory 1010 via a bus 1030, and a database 1050 is used to store data.

[0303] The computing device 1000 also includes an access device 1040, which enables the computing device 1000 to communicate via one or more networks 1060. Examples of these networks include Public Switched Telephone Network (PSTN), Local Area Network (LAN), Wide Area Network (WAN), Personal Area Network (PAN), or combinations of communication networks such as the Internet. The access device 1040 may include one or more of any type of wired or wireless network interface (e.g., Network Interface Controller (NIC)), such as an IEEE 802.11 Wireless Local Area Network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a Universal Serial Bus (USB) interface, a cellular network interface, a Bluetooth interface, or a Near Field Communication (NFC) interface.

[0304] In one embodiment of this specification, the above-described components of the computing device 1000 and Figure 10 Other components, not shown, can also be connected to each other, for example, via a bus. It should be understood that... Figure 10 The block diagram of the computing device shown is for illustrative purposes only and is not intended to limit the scope of this specification. Those skilled in the art can add or replace other components as needed.

[0305] The computing device 1000 can be any type of stationary or mobile computing device, including mobile computers or mobile computing devices (e.g., tablet computers, personal digital assistants, laptop computers, notebook computers, netbooks, etc.), mobile phones (e.g., smartphones), wearable computing devices (e.g., smartwatches, smart glasses, etc.) or other types of mobile devices, or stationary computing devices such as desktop computers or personal computers (PCs). The computing device 1000 can also be a mobile or stationary server.

[0306] The processor 1020 is used to execute the following computer program / instruction, which, when executed by the processor, implements the steps of the above-mentioned data scheduling method or data access method.

[0307] The above is an illustrative scheme of a computing device according to this embodiment. It should be noted that the technical solution of this computing device belongs to the same concept as the technical solutions of the data scheduling method and data access method described above. For details not described in detail in the technical solution of the computing device, please refer to the description of the technical solutions of the data scheduling method or data access method described above.

[0308] An embodiment of this specification also provides a computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the above-described data scheduling method or data access method.

[0309] The above is an illustrative scheme of a computer-readable storage medium according to this embodiment. It should be noted that the technical solution of this storage medium belongs to the same concept as the technical solutions of the data scheduling method and data access method described above. For details not described in detail in the technical solution of the storage medium, please refer to the description of the technical solutions of the data scheduling method or data access method described above.

[0310] An embodiment of this specification also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of the above-described data scheduling method or data access method.

[0311] The above is an illustrative scheme of a computer program product according to this embodiment. It should be noted that the technical solution of this computer program product belongs to the same concept as the technical solutions of the data scheduling method and data access method described above. For details not described in detail in the technical solution of the computer program product, please refer to the description of the technical solutions of the data scheduling method or data access method described above.

[0312] The foregoing has described specific embodiments of this specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recited in the claims may be performed in a different order than that shown in the embodiments and may still achieve the desired result. Furthermore, the processes depicted in the drawings do not necessarily require the specific or sequential order shown to achieve the desired result. In some embodiments, multitasking and parallel processing are possible or may be advantageous.

[0313] The computer instructions include computer program code, which may be in the form of source code, object code, executable file, or some intermediate form. The computer-readable medium may include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium may be appropriately added or removed according to the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0314] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments in this specification are not limited to the described order of actions, because according to the embodiments in this specification, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to the embodiments in this specification.

[0315] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0316] The preferred embodiments disclosed above are merely illustrative of this specification. The optional embodiments do not exhaustively describe all details, nor do they limit the invention to the specific implementations described. Clearly, many modifications and variations can be made based on the embodiments described herein. These embodiments are selected and specifically described in this specification to better explain the principles and practical applications of the embodiments, thereby enabling those skilled in the art to better understand and utilize this specification. This specification is limited only by the claims and their full scope and equivalents.

Claims

1. A data scheduling method, applied to the data scheduling terminal of a cloud storage platform, comprising: Obtain historical usage data of resources on at least one cloud storage service cluster, as well as historical access data of multiple storage containers on the at least one cloud storage service cluster; Based on the historical usage data and the historical access data, data analysis is performed to predict the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within a target time period; Based on the matching relationship, the multiple storage containers are deployed on the corresponding target cloud storage service cluster.

2. The method according to claim 1, wherein the step of performing data analysis based on the historical usage data and the historical access data to predict the matching relationship between the at least one cloud storage service cluster and the plurality of storage containers within a target time period includes: Data analysis is performed on the historical usage data to obtain the predicted usage data of resources on each cloud storage service cluster within the target time period, and data analysis is performed on the historical access data to obtain the predicted access data of each storage container within the target time period. Based on the predicted usage data and the predicted access data, simulations are performed to determine the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within the target time period.

3. The method according to claim 2, wherein the step of performing data analysis on the historical usage data to obtain predicted usage data of resources on each cloud storage service cluster within a target time period includes: By integrating the historical usage data into time-series data, a resource usage profile of resources on each cloud storage service cluster is obtained. Based on the resource usage profiles of the resources on each cloud storage service cluster, machine learning models are used to predict the predicted usage data of the resources on each cloud storage service cluster within a target time period. The step of performing data analysis on the historical access data to obtain predicted access data for each storage container within the target time period includes: The historical access data is integrated with time-series data to obtain the container access profile of each storage container; Based on the container access profiles of each storage container, a machine learning model is used to predict the access data of each storage container within the target time period.

4. The method according to claim 2, wherein the step of performing simulation based on the predicted usage data and the predicted access data to determine the matching relationship between the at least one cloud storage service cluster and the plurality of storage containers within the target time period includes: Under the premise of meeting the preset constraints of the scheduling decision indicators, simulation is performed based on the predicted usage data and the predicted access data to determine the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within the target time period. The scheduling decision indicators include at least one of cluster performance, resource utilization, resource availability and load balancing.

5. The method according to any one of claims 1-4, further comprising, before deploying the plurality of storage containers on the corresponding target cloud storage service cluster based on the matching relationship: Based on the matching relationship, it is determined whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period; The step of deploying the storage container on the corresponding cloud storage service cluster based on the matching relationship includes: If the conditions are met, the multiple storage containers will be deployed on the corresponding target cloud storage service cluster based on the matching relationship.

6. The method according to claim 5, after determining whether each cloud storage service cluster meets the resource requirements of the corresponding storage container within the target time period based on the matching relationship, further comprising: If the requirements are not met, resource maintenance will be performed on the resources on the first target cloud storage service cluster, wherein the first target cloud storage service cluster is the target storage cloud service cluster that does not meet the resource requirements of the corresponding storage container.

7. The method according to claim 6, wherein the cloud storage platform further includes a resource pool; The resource operation and maintenance of the resources on the first target cloud storage service cluster includes: Determine whether the resource pool has any idle resources during the target time period; If so, based on the available resources, the first target cloud storage service cluster is expanded.

8. The method according to claim 7, further comprising, after determining whether the resource pool has idle resources within the target time period: If not, replenish the resource pool. Based on the supplemented resource pool, the resources of the first target cloud storage service cluster are expanded.

9. The method according to claim 1, after deploying the plurality of storage containers on the corresponding target cloud storage service cluster based on the matching relationship, further comprising: If the deployment fails, update the historical usage data of the resources on the second target cloud storage service cluster, and return to the step of performing data analysis based on the historical usage data and the historical access data to predict the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within the target time period, wherein the second target cloud storage service cluster is the target storage cloud service cluster that failed to deploy.

10. The method according to claim 1, wherein any cloud storage service cluster includes a front-end machine group and a cloud storage back-end cluster; The acquisition of historical usage data for resources on at least one cloud storage service cluster includes: Obtain historical usage data of resources on at least one front-end machine group and historical usage data of resources on at least one cloud storage back-end cluster; The step of performing data analysis based on the historical usage data and the historical access data to predict the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within a target time period includes: Based on the historical usage data of the front end and the historical usage data of the back end, data analysis is performed to predict the matching relationship between the at least one front end machine group, the at least one cloud storage back end cluster, and the multiple storage containers within the target time period; The step of deploying the multiple storage containers on the corresponding target cloud storage service cluster based on the matching relationship includes: Based on the matching relationship, the multiple storage containers are deployed on the corresponding target front-end machine group and the target cloud storage back-end cluster.

11. A data access method applied to a target cloud storage service cluster of a cloud storage platform, wherein, The target cloud storage service cluster deploys corresponding target storage containers. The matching relationship between the target storage containers and the target cloud storage service cluster is predicted based on data analysis of historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on the at least one cloud storage service cluster. The method includes: In response to a data access request sent by the terminal within the target time period, data access is performed on the object data in the target storage container.

12. A cloud storage platform, comprising a data scheduling terminal and at least one cloud storage service cluster; The data scheduling terminal is used to obtain historical usage data of resources on at least one cloud storage service cluster and historical access data of multiple storage containers on the at least one cloud storage service cluster. Based on the historical usage data and the historical access data, data analysis is performed to predict the matching relationship between the at least one cloud storage service cluster and the multiple storage containers within a target time period. Based on the matching relationship, the multiple storage containers are deployed on the corresponding target cloud storage service cluster. The at least one cloud storage service cluster is used to access object data in the deployed storage container in response to data access requests sent by the terminal within the target time period.

13. A computing device, comprising: Memory and processor; The memory is used to store computer programs / instructions, and the processor is used to execute the computer programs / instructions, which, when executed by the processor, implement the steps of the method according to any one of claims 1 to 11.

14. A computer-readable storage medium storing a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

15. A computer program product comprising a computer program / instructions that, when executed by a processor, implement the steps of the method according to any one of claims 1 to 11.

Images