Resource scheduling method and apparatus

Abstract

Provided in the embodiments of the present disclosure are a resource scheduling method and apparatus. The resource scheduling method comprises: determining historical operating log data and historical resource usage data of a data processing node set which are generated within a preset historical time period; on the basis of the historical operating log data, and by means of using a log prediction model, predicting predicted operating log data of the data processing node set that is generated within a preset future time period; on the basis of the predicted operating log data, and by means of using a resource prediction model, predicting predicted resource usage data of the data processing node set that is required within the preset future time period; on the basis of the historical resource usage data, determining target resource usage data of the data processing node set within the preset future time period; and on the basis of the predicted resource usage data and the target resource usage data, performing resource scheduling on the data processing node set. The resource scheduling method is more reasonable and more intelligent, and does not require human participation, thus resulting in more stable application systems.

Description

[0001] Resource Scheduling Method and Apparatus This disclosure claims priority to Chinese patent application number 202410115618.6, filed with the China Patent Office on January 26, 2024, entitled "Resource Scheduling Method and Apparatus," the entire contents of which are incorporated herein by reference. Technical Field: Embodiments of the present disclosure relate to the field of computer technology, and more particularly, to a resource scheduling method. One or more embodiments of the present disclosure also relate to a resource scheduling apparatus, a computing device, a computer-readable storage medium, and a computer program product. Background: Typically, cloud storage systems utilize multiple cloud storage devices or physical storage devices to collaboratively provide data access. In practical applications, cloud storage systems typically divide multiple cloud storage devices or physical storage devices into multiple groups based on actual needs. Each group corresponds to different needs and provides different data access services. To prevent certain groups from running out of data processing resources when providing their corresponding data access services, one solution is to implement water level monitoring and alarms for each group, regularly monitor each group's water resource usage, and manually expand capacity for groups that trigger water level monitoring alarms. However, this solution often leads to a mismatch between the resources required by the groups and the resources allocated to them. For example, some groups may have high resource demands but insufficient resources, while others may have low resource demands and waste resources. This situation results in irrational resource allocation in the cloud storage system. Furthermore, manual capacity expansion is often not timely enough, which can easily lead to serious service failures and unstable service provision in the cloud storage system. Therefore, a more reasonable and intelligent resource scheduling method is urgently needed to address the above technical issues. SUMMARY OF THE INVENTION In view of this, embodiments of the present disclosure provide a resource scheduling method. One or more embodiments of the present disclosure also relate to a resource scheduling apparatus, a computing device, a computer-readable storage medium, and a computer program product to address the technical deficiencies existing in the prior art.According to a first aspect of an embodiment of the present disclosure, a resource scheduling method is provided, comprising: determining historical operation log data and historical resource usage data generated by a data processing node set within a preset historical time period; predicting, based on the historical operation log data, predicted operation log data generated by the data processing node set within a preset future time period using a log prediction model; predicting, based on the predicted operation log data, predicted resource usage data required by the data processing node set within the preset future time period using a resource prediction model; determining, based on the historical resource usage data, target resource usage data of the data processing node set within the preset future time period; and performing resource scheduling on the data processing node set based on the predicted resource usage data and the target resource usage data. According to a second aspect of an embodiment of the present disclosure, a resource scheduling device is provided, comprising: a historical data determination module, configured to determine historical operation log data and historical resource usage data generated by a data processing node set within a preset historical time period; a future operation log data prediction module, configured to predict the predicted operation log data generated by the data processing node set within a preset future time period using a log prediction model based on the historical operation log data; a future resource usage data prediction module, configured to predict the predicted resource usage data required by the data processing node set within the preset future time period based on the predicted operation log data using a resource prediction model; a target resource usage data determination module, configured to determine the target resource usage data of the data processing node set within the preset future time period based on the historical resource usage data; and a resource scheduling module, configured to perform resource scheduling on the data processing node set based on the predicted resource usage data and the target resource usage data. According to a third aspect of an embodiment of the present disclosure, a computing device is provided, comprising: a memory and a processor; the memory is configured to store computer-executable instructions, and the processor is configured to execute the computer-executable instructions. When executed by the processor, the computer-executable instructions implement the steps of the resource scheduling method described above. According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, storing computer-executable instructions. When executed by the processor, the instructions implement the steps of the resource scheduling method described above. According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising a computer program / instructions. When executed by the processor, the computer program / instructions implement the steps of the resource scheduling method described above.An embodiment of the present disclosure provides a resource scheduling method, comprising: determining historical operation log data and historical resource usage data generated by a data processing node set within a preset historical time period; predicting, based on the historical operation log data, predicted operation log data generated by the data processing node set within a preset future time period using a log prediction model; predicting, based on the predicted operation log data, predicted resource usage data required by the data processing node set within the preset future time period using a resource prediction model; determining, based on the historical resource usage data, target resource usage data of the data processing node set within the preset future time period; and performing resource scheduling on the data processing node set based on the predicted resource usage data and the target resource usage data. Specifically, the method utilizes historical operation log data of a data processing node set (i.e., a group of the aforementioned multiple cloud storage devices or physical devices) to predict future predicted operation log data using a log prediction model, thereby enabling more realistic and reasonable resource scheduling. Furthermore, based on the resource prediction model's ability to analyze the relationship between operation log data and resource usage data, as well as the predicted and future predicted operation log data, predicted resource usage data for the data processing node set is predicted. When the predicted operation log data is more realistic and reasonable, the accuracy of the predicted resource usage data is further improved. This allows resource scheduling for scaling up or down the data processing nodes based on the predicted resource usage data, which represents future resources required by the data processing node set, and the target resource usage data, which represents the capabilities currently provided by the data processing node set. This improves data processing node resource utilization and makes data processing node resource scheduling more reasonable. Furthermore, the entire resource scheduling process is implemented through computer intelligent analysis and scheduling, eliminating the need for human intervention. This ensures timely resource scheduling and stable service provision for the system that utilizes the data processing node set for data processing. BRIEF DESCRIPTION OF THE DRAWINGS FIG1 is a block diagram illustrating a specific application scenario of a resource scheduling method according to an embodiment of the present disclosure; FIG2 is a flow chart illustrating a resource scheduling method according to an embodiment of the present disclosure; FIG3 is a flow chart illustrating a processing process of a resource scheduling method according to an embodiment of the present disclosure; FIG4 is a schematic diagram illustrating a resource scheduling apparatus according to an embodiment of the present disclosure; and FIG5 is a block diagram illustrating a computing device according to an embodiment of the present disclosure. The following description sets forth numerous specific details to facilitate a thorough understanding of the present disclosure.However, the present disclosure can be implemented in many other ways than those described herein, and those skilled in the art can make similar generalizations without departing from the scope of the present disclosure. Therefore, the present disclosure is not limited to the specific embodiments disclosed below. The terminology used in one or more embodiments of the present disclosure is for the purpose of describing specific embodiments only and is not intended to limit the present disclosure. As used in one or more embodiments of the present disclosure and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" used in one or more embodiments of the present disclosure refers to and encompasses any and all possible combinations of one or more of the associated listed items. It should be understood that while the terms "first," "second," and so on may be used to describe various information in one or more embodiments of the present disclosure, such information should not be limited to these terms. These terms are used solely to distinguish information of the same type from one another. For example, first could be referred to as second, and similarly, second could be referred to as first, without departing from the scope of one or more embodiments of the present disclosure. Depending on the context, the term "if" as used herein can be interpreted as meaning "when," "when," or "in response to determining." Furthermore, it should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, storage, and display) involved in one or more embodiments of this disclosure are all authorized by the user or fully authorized by all parties. The collection, use, and processing of such data must comply with the relevant laws, regulations, and standards of the relevant region, and corresponding access points are provided for users to choose to authorize or deny such data. First, the terms used in one or more embodiments of this disclosure are explained.

[0002] DNS: Domain Name System, is a service of the Internet. As a distributed database that maps domain names and IP addresses to each other, it enables people to access the Internet more conveniently.

[0003] VIP: virtual IP, a virtual IP (Internet Protocol) address used to provide services in a cloud server cluster.

[0004] QPS: Queries Per Second, represents the number of queries or requests a system can process per unit of time (usually per second). It can be used to measure system throughput and response speed. Generally, a higher QPS indicates a greater load the system can handle and better performance. Region: The region where the cloud storage data center is located. Bucket: The data management unit used by users to access cloud storage. In object storage systems, a bucket can be understood as a storage bucket used to organize and store data objects. Group: Multiple cloud servers and / or physical devices that provide the same service and work together can form a group. In cloud storage scenarios, front-end resources are typically organized into groups, providing services based on the application requirements of each group. The following detailed explanation will be given using the resource scheduling method provided in the embodiments of the present disclosure as an example of a cloud storage scenario. It should be noted that the resource scheduling method provided in the embodiments of the present disclosure is not limited to cloud storage scenarios and can also be applied to other application scenarios with resource scheduling requirements, such as physical server clusters and distributed systems. For other application scenarios, please refer to the embodiments of the present disclosure and will not be further described. With the rapid development of cloud storage, user data is being migrated to the cloud in large quantities. Cloud storage front-end resources are responsible for uploading, downloading, and processing user data. Utilizing the maximum read and write capabilities of these front-end resources presents a challenge. Traditional solutions typically divide a region's front-end resources into multiple front-end groups to handle different data processing tasks. For example, front-end resources are divided into intranet and extranet front-end groups based on traffic sources to handle both intranet and extranet requests. Alternatively, front-end resource groups are divided into regular customer and VIP customer front-end groups based on user demographics to handle data processing requests from different customer groups. Water level monitoring and alarms are used to regularly monitor the resources in each front-end group to see if they have triggered alarm thresholds. If so, the capacity of the triggering front-end group is manually expanded. However, these traditional solutions have the following drawbacks:

[0005] (1) When the resources of the front-end group are tight and capacity expansion is required, the solution of water level monitoring and alarm is relatively passive. When a front-end group triggers an alarm, timely capacity expansion is required, but manual capacity expansion cannot be carried out in time, which will cause the online service of the front-end group to be in danger;

[0006] (2) When the machine resource utilization rate of a front-end group is low, resources will be wasted, and it will be impossible to maximize the utilization of each front-end group. In view of this, the present disclosure provides a resource scheduling method. The present disclosure also relates to a resource scheduling device, a computing device, a computer-readable storage medium, and a computer program product to improve the front-end resource utilization capability of the entire region and provide users with cloud storage services with better performance. Detailed descriptions are given one by one in the following embodiments. The following, in conjunction with Figure 1, takes the application of the resource scheduling method provided by the present disclosure in a cloud storage system as an example to further illustrate the resource scheduling method provided by the present disclosure. See Figure 1, which is a structural block diagram of a specific application scenario of a resource scheduling method provided according to an embodiment of the present disclosure. As shown in FIG1 , FIG1 includes a cloud storage system 102 and a resource scheduling server 104. The cloud storage system 102 can be a public cloud storage system, a private cloud storage system, or a hybrid cloud storage system. The resource scheduling server 104 can be an independent physical server or a server cluster consisting of multiple physical servers. Alternatively, the resource scheduling server 104 can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms. This disclosure is not limited to this aspect. The front-end resource pool includes multiple physical storage devices or cloud storage devices. The machines (i.e., physical storage devices or cloud storage devices) in the front-end resource pool can be divided into resources in the intranet front-end group and resources in the extranet front-end group based on traffic sources. In specific implementation, when a user uses the cloud storage system 102 for data processing, he first accesses the cloud storage system 102 on the Internet through the DNS and sends a data processing request to the cloud storage system 102. After receiving the user's data processing request, the cloud storage system 102 can determine the front-end resource group for data processing based on the traffic source of the user's data processing, and allocate the user's data processing request to the corresponding front-end resource group through the VIP. Thereafter, the front-end resource group parses the data processing request and schedules the cloud storage back-end cluster to complete the data processing.In this process, since resources of the intranet front-end group and the extranet front-end group can be scheduled using the resource scheduling method provided in the embodiments of the present disclosure, specifically, the resource scheduling server 104 determines a bucket profile corresponding to a bucket accessed by each front-end group (i.e., a set of data processing nodes). The bucket profile is quantified as historical operation log data generated by the buckets in each front-end group within a preset historical time period (e.g., two weeks) when the user processes data through each front-end group. The resource scheduling server 104 also determines historical resource usage data corresponding to each front-end group. Subsequently, the log prediction model is used to predict the bucket profiles of each front-end group in the future. Based on the bucket profiles of each front-end group in the future, the resource prediction model is used to predict the resource usage data required by each front-end group within a preset future time period (e.g., two weeks). Based on the predicted resource usage data required by each front-end group and the target resource usage data, data processing node resource scheduling is performed for each front-end group, with a predicted expansion or contraction prediction. This implements resource rotation among the front-end groups, thereby improving resource utilization of the data processing nodes in the front-end resource pool. The resource scheduling method provided by the embodiments of the present disclosure uses a log prediction model to analyze historical operation log data and predict future operation log data, making subsequent resource scheduling more realistic and reasonable. Furthermore, based on the resource prediction model's ability to analyze the relationship between operation log data and resource usage data, and the predicted and future predicted operation log data, the predicted resource usage data for each front-end group is predicted. This further improves the accuracy of the predicted resource usage data for each front-end group, making resource scheduling for capacity expansion or contraction more accurate based on the predicted resource usage data and target resource usage data required by each front-end group. This achieves reasonable and accurate resource scheduling. Furthermore, the entire resource scheduling process is implemented through computer intelligent analysis and scheduling, eliminating the need for human intervention. This ensures timely resource scheduling and the stability of services provided by the cloud storage system. Referring to Figure 2, a flow chart of a resource scheduling method provided according to an embodiment of the present disclosure is provided, specifically including the following steps. Step 202: Determine historical operation log data and historical resource usage data generated by a data processing node set within a preset historical time period.Among them, the data processing node set can be understood as a set of one or more data processing nodes. The data processing node can be understood as a node used for data processing, including but not limited to physical devices, or cloud servers that provide basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, and big data and artificial intelligence platforms; for example, in a cloud storage scenario, the data processing node set can be understood as a front-end group. A front-end group consists of multiple machines (cloud servers, physical devices, etc.), and the data processing node can be understood as the machine corresponding to the front-end group. The preset historical time period can be understood as a preset time period of a fixed length. For example, the fixed length can be understood as a fixed length, i.e., 1 hour, five days, two weeks, etc. When the fixed length is understood as 1 day, the preset historical time period can be October 1, 2023, October 2, 2023, October 5, 2023, etc.; or, when the fixed length is understood as 1 week, the preset historical time period can be understood as October 1, 2023 to October 7, 2023, October 15, 2023 to October 22, 2023, etc. Historical operation log data can be understood as a historical bucket profile for a collection of data processing nodes. Bucket profiles are generated by profiling historical bucket read and write operation log data. This data is generated when users instruct front-end groups to process data using buckets within a preset historical time period. Historical operation log data includes, but is not limited to, bucket read and write log data and bucket deletion log data. Historical resource usage data can be understood as historical data on system resource usage by front-end groups within a preset historical time period. Resource usage data includes, but is not limited to, read and write queries per second (QPS), disk utilization, CPU (central processing unit) utilization, memory utilization, and network bandwidth.In actual applications, since the front-end group usually meets the 80 / 20 rule when performing data processing, that is, access to key buckets occupies a higher proportion of resources of the data processing node set, in order to improve resource scheduling speed and ensure resource scheduling accuracy, the access status of the data processing node set to all storage spaces can be characterized by analyzing the access status of key storage spaces. The specific implementation method is as follows: Determining the historical operation log data and historical resource usage data generated by the data processing node set within a preset historical time period includes: determining multiple storage spaces corresponding to the data processing node set, and determining a target storage space from the multiple storage spaces based on the number of accesses by the data processing node set to the multiple storage spaces; and obtaining the historical operation log data and historical resource usage data generated by the data processing node set when accessing the target storage space within the preset historical time period. Here, a storage space can be understood as a bucket, and each data processing node set can have multiple buckets. In a specific implementation, multiple storage spaces corresponding to the data processing node set are determined, and a target storage space is determined from the multiple storage spaces based on the number of accesses to the multiple storage spaces by the data processing node set. This can be understood as determining multiple buckets corresponding to the data processing node set, and then, based on the number of accesses to each bucket by the front-end group, determining one or more buckets with a higher number of accesses as target buckets (i.e., target storage spaces). It should be noted that the specific criteria for determining the target buckets (such as the number of target buckets, or determining a bucket with a number of accesses greater than a certain threshold as the target bucket, and the setting of the threshold, etc.) can be determined according to actual needs and are not limited in the present embodiment. Furthermore, obtaining historical operation log data and historical resource usage data generated by the data processing node set when accessing the target storage space within a preset historical time period can be understood as obtaining historical bucket profiles and historical resource usage data generated by the front-end group when accessing the target bucket within the preset historical time period after determining the target bucket.The resource scheduling method provided by the embodiments of the present disclosure determines a representative target storage space from multiple storage spaces corresponding to a set of data processing nodes based on the number of accesses by the data processing nodes to each storage space. The method then uses historical log data and resource usage data from the data processing nodes' accesses to the target storage space as historical reference data. This improves the processing speed of resource scheduling while ensuring the accuracy of subsequent predictions and resource scheduling based on the historical log data and resource usage data of the target storage space. Step 204: Based on the historical operation log data, a log prediction model is used to predict predicted operation log data generated by the set of data processing nodes within a preset future time period. The log prediction model can be understood as a model for predicting future log data based on historical log data. Preferably, the log prediction model can be understood as a machine learning model used to improve the accuracy of applying the log prediction model to predict future predicted operation log data. For example, the log prediction model can be understood as an ensemble regression model, a recurrent neural network model, or the like. The preset future time period can be understood as a preset future time period of a fixed duration. The practical meaning of the preset future time period is consistent with the preset historical time period, differing only in the historical and future time periods. This will not be elaborated upon. It should be noted that the duration of the preset future time period may or may not be the same as the duration of the preset historical time period, and this is not limited in the present embodiment. In specific implementation, based on historical operation log data, a log prediction model is utilized to predict the predicted operation log data generated by the data processing node set within the preset future time period. This can be understood as inputting the historical operation log data as input to the log prediction model to obtain the output of the log prediction model. This output is the predicted operation log data generated by the data processing node set within the preset future time period. Furthermore, in order to improve the accuracy of the log prediction model and predict future predicted operation log data, the log prediction model can be pre-trained. The specific implementation method is as follows: before predicting the predicted operation log data generated by the data processing node set within a preset future time period based on the historical operation log data and using the log prediction model, it also includes: obtaining a plurality of sample historical operation log data generated within the preset historical time period, and sample predicted operation log data generated within the preset future time period corresponding to each sample historical operation log data; training the initial log prediction model based on the sample historical operation log data and the sample predicted operation log data until the training stop condition is reached to obtain the log prediction model.Among them, multiple preset historical time periods can be understood as multiple preset fixed-length time periods, and multiple preset future time periods can be understood as multiple preset fixed-length time periods corresponding to the multiple preset historical time periods; for example, taking the time period as one day as an example, the multiple preset historical time periods can be understood as October 1, 2023, October 2, 2023, and October 3, 2023, the preset future time period corresponding to October 1, 2023 can be understood as October 2, 2023, the preset future time period corresponding to October 2, 2023 can be understood as October 3, 2023, and the preset future time period corresponding to October 3, 2023 can be understood as October 4, 2023. The initial log prediction model can be understood as an initial, untrained log prediction model. The training stop condition can be understood as satisfying the number of iterations (e.g., 200), meeting the preset accuracy requirement, and the error being below a preset threshold. In actual applications, the training stop condition can be set based on application requirements and is not limited in this regard in the presently disclosed embodiments. Therefore, obtaining sample historical operation log data generated within multiple preset historical time periods and sample predicted operation log data generated within a preset future time period corresponding to each sample historical operation log data can be understood as obtaining multiple sample historical operation log data by counting bucket profiles generated by front-end groups within multiple preset historical time periods, and obtaining multiple sample predicted operation log data by counting bucket profiles generated by front-end groups within multiple corresponding preset future time periods. Furthermore, after obtaining sample historical operation log data generated within multiple preset historical time periods and sample predicted operation log data generated within a preset future time period corresponding to each sample historical operation log data, an initial log prediction model is trained using the sample historical operation log data generated within the multiple preset historical time periods as samples and the sample labels of the sample predicted operation log data generated within the preset future time period corresponding to each sample historical operation log data until a training stop condition is met. Upon completion of the training, a log prediction model is obtained. The resource scheduling method provided in the disclosed embodiments trains a log prediction model using sample historical operation log data generated within multiple preset historical time periods and sample predicted operation log data generated within the preset future time period corresponding to each sample historical operation log data, thereby ensuring that the prediction results of the log prediction model in subsequent applications are more accurate and more consistent with actual conditions.Step 206: Based on the predicted operation log data, use a resource prediction model to predict the predicted resource usage data required by the data processing node set within the preset future time period. The resource prediction model is a model for predicting resource usage data based on log data. Optionally, the resource prediction model can be understood as a machine learning model used to improve the accuracy of applying the resource prediction model to predict future predicted resource usage data required within the preset future time period. For example, the log prediction model can be understood as a linear regression model, an ensemble regression model, a recurrent neural network model, etc. Furthermore, to improve the accuracy of the log prediction model and future predicted operation log data, a resource prediction model can be pre-trained. The specific implementation is as follows: Before predicting the predicted resource usage data required by the data processing node set within the preset future time period using the resource prediction model based on the predicted operation log data, the method further includes: obtaining multiple sample historical operation log data generated within the preset historical time period, and sample historical resource usage data corresponding to each sample historical operation log data; and training an initial resource prediction model based on the sample historical operation log data and the sample historical resource usage data until a training stop condition is met, thereby obtaining a resource prediction model. The sample historical resource usage data corresponding to each sample historical operation log data can be understood as the system resource usage data within the historical time period corresponding to each sample historical operation log data. For specific implementation, the specific implementation of resource prediction model training can be referred to the embodiments described above. The only difference lies in the sample labels used during model training: the sample labels used for training the resource prediction model are the sample historical resource usage data corresponding to each sample historical operation log data, i.e., the system resource usage data of the front-end group corresponding to each sample historical operation log data. The resource scheduling method provided in the embodiments of the present disclosure trains a resource prediction model using sample historical operation log data generated within multiple preset historical time periods, as well as sample historical resource usage data corresponding to each sample historical operation log data. This allows the resource prediction model to more accurately analyze the correlation between the operation log data and resource usage data. This allows for more accurate prediction of resource usage data required by a set of data processing nodes within a preset future time period based on the predicted operation log data in subsequent applications. Step 208: Determine target resource usage data for the set of data processing nodes within the preset future time period based on the historical resource usage data.Target resource usage data can be understood as resource usage data obtained by quantifying historical resource usage data, which can be used to characterize the external service capabilities of a data processing node set. Therefore, determining the target resource usage data for the data processing node set in a preset future time period based on the historical resource usage data can be understood as quantizing the historical resource usage data to obtain quantified resource usage data, and determining the quantized resource usage data as the target resource usage data for the data processing node set in the preset future time period. In practical applications, to ensure the accuracy of the target resource usage data, a reference value can be obtained through offline testing, and the reference value can be adjusted based on online resource usage data to obtain the target resource usage data. This is specifically implemented as follows: Determining the target resource usage data for the data processing node set in the preset future time period based on the historical resource usage data includes: obtaining simulated resource usage data generated by the data processing node set in the preset historical time period; and adjusting the simulated resource usage data based on the historical resource usage data to obtain the target resource usage data for the data processing node set in the preset future time period. Here, simulated resource usage data can be understood as pre-running offline simulation tests on the front-end group to evaluate resource usage data representing the performance of the front-end group itself. In specific implementations, simulation tests can be conducted on each machine in the front-end group using common read / write models through offline testing methods such as stress testing to evaluate the performance of each machine in the group. Since the performance of each machine in the front-end group is similar or identical, the resource usage data that the front-end group can provide can be evaluated based on the performance of each machine and the number of machines. Furthermore, after obtaining simulated resource usage data generated by a set of data processing nodes within a preset historical time period, the simulated resource usage data is fine-tuned based on the historical resource usage data to obtain more accurate target resource usage data for the set of data processing nodes within a preset future time period.The resource scheduling method provided in the embodiments of the present disclosure obtains simulated resource usage data generated by a set of data processing nodes within a preset historical time period, thereby obtaining the theoretical external service capabilities of the set of data processing nodes. The simulated resource usage data is then adjusted using the historical resource usage data to obtain a more realistic external service capability of the set of data processing nodes, namely, more realistic target resource usage data for the set of data processing nodes within the preset future time period. This further improves the accuracy of the target resource usage data and the accuracy of subsequent resource scheduling using the target resource usage data as a reference. Step 210: Perform resource scheduling on the set of data processing nodes based on the predicted resource usage data and the target resource usage data. Resource scheduling can be understood as scaling up or down the set of data processing nodes. During specific implementation, in order to improve the resource utilization of data processing nodes, resource scheduling can be performed on multiple data processing node sets, and the specific implementation method is as follows: the data processing node set includes at least two; the resource scheduling of the data processing node set based on the predicted resource usage data and the target resource usage data includes: determining the predicted resource usage data and target resource usage data of each data processing node set in the preset future time period; determining the decision strategy of each data processing node set based on the difference between the predicted resource usage data of each data processing node set in the preset future time period and the target resource usage data; and expanding or shrinking the capacity of each data processing node set according to the decision strategy to achieve resource scheduling for each data processing node set. The difference between the predicted resource usage data and the target resource usage data can be understood as the difference between the resource usage data for each dimension in the predicted resource usage data and the resource usage data for the corresponding dimension in the target resource usage data. For example, the difference between the quantized value of CPU (Central Processing Unit) utilization in the predicted resource usage data and the quantized value of CPU utilization in the target resource usage data, the difference between the quantized value of memory utilization in the predicted resource usage data and the quantized value of memory utilization in the target resource usage data, etc. The decision strategy can be understood as a strategy for scaling up, scaling down, or not scheduling a set of data processing nodes determined based on the short board difference (i.e., a large difference or a large number of machines corresponding to the difference) between the resource usage data for each dimension in the predicted resource usage data and the resource usage data for the corresponding dimension in the target resource usage data.Once the decision policy is determined, the data processing nodes can be expanded or reduced in capacity based on the policy for scaling up, scaling down, or not scheduling each data processing node set as specified in the decision policy. The resource scheduling method provided in the embodiments of the present disclosure determines a decision policy for whether each data processing node set needs to be scaled up or down based on the difference between the predicted resource usage data for each data processing node set within a preset future time period and the target resource usage data. The data processing nodes are then scaled up or down based on the decision policy, achieving automated resource scheduling for each data processing node set. Furthermore, since no manual intervention is required, service failures caused by manual scaling are reduced, thereby improving the stability of the services provided by each data processing node set. In practical applications, to improve resource utilization, capacity expansion or contraction can be implemented specifically based on the number of nodes to be expanded and / or the number of nodes to be contracted. The specific implementation is as follows: Expanding or contracting each set of data processing nodes based on the decision policy includes: determining, based on the decision policy, a set of data processing nodes to be expanded and / or a set of data processing nodes to be contracted; determining the number of nodes to be expanded in the set of data processing nodes to be expanded and / or the number of nodes to be contracted in the set of data processing nodes to be contracted; and expanding or contracting each set of data processing nodes based on the number of nodes to be expanded and / or the number of nodes to be contracted. The number of nodes to be expanded can be understood as the number of machines required to be expanded for the front-end group to be expanded, and the number of nodes to be contracted can be understood as the number of machines required to be contracted for the front-end group to be contracted. After the decision strategy is determined, the front-end groups that need to be expanded or reduced can be determined based on the decision strategy, and the corresponding number of expansion machines and the number of reduction machines can be determined. Thus, each data processing node can be expanded or reduced based on the number of expansion machines and / or the number of reduction machines. The resource scheduling method provided in the embodiments of the present disclosure determines, based on the decision strategy, the data processing node sets that need to be expanded and / or the data processing node sets that need to be reduced in the data processing node set, and expands or reduces each data processing node set based on the corresponding number of expansion nodes and / or the number of reduction nodes. This allows resource scheduling to accurately expand and / or reduce each data processing node set, thereby achieving higher resource utilization and reducing resource waste.In practical applications, to ensure the accuracy of expansion and / or contraction, expansion and / or contraction can be performed in three scenarios, specifically implemented as follows: Expanding or contracting each data processing node set based on the number of expanded nodes and / or the number of contracted nodes includes: expanding each data processing node set based on the number of expanded nodes; or contracting each data processing node set based on the number of contracted nodes; or expanding or contracting each data processing node set based on both the number of expanded nodes and the number of contracted nodes. Expanding each data processing node set based on the number of expanded nodes can be understood as a scenario where all data processing node sets require expansion, i.e., each data processing node set is expanded based on the number of expanded nodes corresponding to each data processing node set. Contracting each data processing node set based on the number of contracted nodes can be understood as a scenario where all data processing node sets require contraction, i.e., each data processing node set is contracted based on the number of contracted nodes corresponding to each data processing node set. Expanding or shrinking each data processing node set based on the number of nodes to be expanded or shrunk can be understood as including both data processing node sets that require expansion and data processing node sets that require shrinking. In this case, the data processing node sets that require expansion need to be expanded, while the data processing node sets that require shrinking need to be shrunk. The resource scheduling method provided in the embodiments of the present disclosure considers the actual need for expansion or shrinking of each data processing node set during resource scheduling for each data processing node set, thereby implementing resource scheduling for expansion or shrinking of each data processing node set, ensuring accurate resource scheduling. In practical applications, resource utilization of data processing nodes can be improved by rotating resources among various data processing node sets. The specific implementation method is as follows: the expansion or reduction of each data processing node set according to the number of expanded nodes and the number of reduced nodes includes: reducing the data processing nodes in the data processing node set to be reduced to the idle node set according to the number of reduced nodes; and increasing the reduced idle data processing nodes in the idle node set to the data processing node set to be expanded according to the number of expanded nodes, wherein the idle data processing nodes are any one or more data processing nodes in the data processing node set to be reduced.The idle node set can be understood as an idle machine pool. The idle machine pool contains the machines to be scaled down by the data processing node set to be scaled down. After the data processing node set to be scaled down has scaled down the machines to the idle machine pool, the data processing node set to be expanded can be expanded using the machines that were scaled down from the data processing node set to be scaled down and moved to the idle machine pool. In specific implementation, when resource scheduling determined according to the decision policy described above includes both the data processing node set to be expanded and the data processing node set to be scaled down, the data processing nodes in the data processing node set to be scaled down can first be placed into the idle node set based on the number of nodes to be scaled down. Then, based on the number of nodes to be expanded, data processing nodes in the data processing node set to be scaled down can be removed from the idle node set as many as the number of nodes to be expanded and added to the data processing node set to be expanded. The resource scheduling method provided by the embodiments of the present disclosure, when there are both data processing node sets that need to be expanded and data processing node sets that need to be reduced in capacity, rotates resources from the data processing node sets that need to be reduced to the data processing node sets that need to be expanded through idle node sets. This improves resource utilization in each data processing node set, reduces the probability of failure of a data processing node set when providing services, and enhances the user experience. In actual applications, there may be a situation where the data processing nodes in the idle node set are sufficient for capacity expansion. In this case, the resource scheduling server can directly extract data processing nodes equal to the number of expansion nodes from the idle node set based on the number of expansion nodes, and expand the data processing nodes to the data processing node set to be expanded. The specific implementation method is as follows: expanding the idle data processing nodes that have been scaled down in the idle node set to the data processing node set to be expanded based on the number of expansion nodes includes: determining the number of idle nodes in the idle node set that have been scaled down; if it is determined that the number of expansion nodes is less than or equal to the number of idle nodes, determining a target idle data processing node from the idle data processing nodes based on the number of expansion nodes; and expanding the target idle data processing node to the data processing node set to be expanded. The number of idle nodes is equal to the sum of the number of scaled down nodes in all data processing node sets that need to be scaled down.If the number of nodes being expanded is less than or equal to the number of idle nodes, that is, the number of data processing nodes being expanded is less than or equal to the number of data processing nodes being reduced, resource rotation can be performed directly. Based on the number of nodes being expanded, target idle data processing nodes equal in number to the number of nodes to be expanded are determined from the idle data processing nodes, and the target idle data processing nodes are then expanded to the set of data processing nodes to be expanded. The resource scheduling method provided in the embodiments of the present disclosure directly performs resource rotation when there are sufficient data processing nodes in the set of idle nodes for expansion. Specifically, the idle data processing nodes that have been reduced are expanded to the set of data processing nodes to be expanded, further improving resource utilization within each set of data processing nodes. In actual applications, there may be situations where the data processing nodes in the idle node set are insufficient for capacity expansion. In this case, data processing nodes can be added from a third-party platform for capacity expansion. The specific implementation method is as follows: After determining the number of idle data processing nodes in the idle node set after the reduction, the method further includes: if it is determined that the number of expanded nodes is greater than the number of idle nodes, determining the difference between the number of expanded nodes and the number of idle nodes; determining expansion data processing nodes from the third-party platform based on the difference; and expanding the idle data processing nodes and the expansion data processing nodes to the data processing node set to be expanded. The third-party platform can be understood as a third-party platform that can provide cloud devices, physical devices, etc. For example, the third-party platform can be understood as an offline machine provision platform, an online server provision platform, etc. In specific implementations, if the number of expansion nodes exceeds the number of idle nodes, the difference between the number of expansion nodes and the number of idle nodes can be used as the number of expansion data processing nodes. A corresponding number of expansion data processing nodes can then be determined from a third-party platform. Subsequently, the idle and expansion data processing nodes are added to the set of data processing nodes to be expanded. The resource scheduling method provided in the embodiments of the present disclosure uses a third-party platform to add expansion data processing nodes when the number of expansion nodes exceeds the number of idle nodes, thereby ensuring normal expansion of the set of data processing nodes to be expanded and ensuring data processing stability for the set of data processing nodes.The resource scheduling method provided by the embodiments of the present disclosure utilizes historical operation log data of a data processing node set (i.e., a group of the aforementioned multiple cloud storage devices or physical devices) to predict future predicted operation log data using a log prediction model, thereby enabling more realistic and reasonable resource scheduling. Furthermore, based on the resource prediction model's ability to analyze the relationship between operation log data and resource usage data, as well as the predicted and future predicted operation log data, predicted resource usage data for the data processing node set is predicted in the future. When the predicted operation log data is more realistic and reasonable, the accuracy of the predicted resource usage data is further improved. This allows resource scheduling for scaling data processing nodes based on the predicted resource usage data, which represents future resources required by the data processing node set, and the target resource usage data, which represents the capabilities currently provided by the data processing node set. This improves data processing node resource utilization and makes data processing node resource scheduling more reasonable. Furthermore, the entire resource scheduling process is implemented through computer intelligent analysis and scheduling, eliminating the need for human intervention. This ensures timely resource scheduling for a system that uses the data processing node set for data processing. and the stability of the services provided by the system. The following, in conjunction with Figure 3, further illustrates the resource scheduling method provided by the present disclosure, using its application in a cloud storage system as an example. Figure 3 is a flowchart of the processing process of a resource scheduling method provided according to an embodiment of the present disclosure. The specific process is described below.

[0007] (1) Bucket reading and writing When users use the cloud storage system for data processing, they upload and download data through buckets, and send bucket reading and writing data processing requests to the cloud storage system for the data stored in the bucket, so that the cloud storage responds to the data processing requests.

[0008] (2) Front-end resource grouping: Based on actual application requirements, front-end resources for data processing are pooled and managed, that is, front-end resources are divided into multiple front-end groups. For example, according to the source of traffic, the resources in the front-end resource pool are divided into intranet front-end groups (i.e., cluster 1 in Figure 3) and extranet front-end groups (i.e., cluster 2 in Figure 3). The intranet front-end groups provide intranet capabilities, which can provide users with data transmission and data processing services within the private network of the cloud service (cloud service provided by the cloud storage system). The extranet front-end groups provide extranet capabilities, which can provide users with data transmission and processing on the public network, realizing communication between the cloud service and the public network. Then, when a user uses the cloud storage system for data processing, the cloud storage system can determine the corresponding front-end resource group that provides the service based on the user's data processing request. The corresponding front-end resource group processes the data processing request and generates raw bucket read and write operation logs (data 1 in Figure 3), as well as statistical system performance data for each front-end resource group (data 2 in Figure 3). It should be noted that there are multiple buckets in a front-end group, but the bucket usage of the front-end group resources generally meets the 80 / 20 principle. The vast majority of resources are used by a small number of buckets. Therefore, one or more key buckets that occupy a large amount of resources can be identified. These key buckets largely determine the usage trend of the front-end group resources.

[0009] (3) Front-end resource capability assessment

[0010] 1. Conduct offline simulation tests on each machine in each front-end group to obtain the performance reference value of each front-end group (i.e. the above-mentioned simulated resource usage data).

[0011] 2. Collect system performance data for each front-end group over the past two weeks, such as read and write queries per second (QPS), disk utilization, CPU utilization, memory utilization, and network bandwidth. Quantify the system performance data for each front-end group through QPS analysis, disk analysis, CPU analysis, memory analysis, and bandwidth analysis, and calculate the quantified system performance data for each front-end group. For example, CPU analysis can quantify the number of CPU cores consumed for a particular front-end group. For example, on a 96-core CPU machine, 40% CPU utilization indicates 38 CPU cores consumed. Memory analysis can quantify the memory utilization for a particular front-end group. For example, on a machine with 128GB of memory, 20% memory utilization indicates 25.6GB of memory consumed. Disk analysis can quantify the disk read and write IOPS for a particular front-end group.

[0012] (Input / Output Operations Per Second, input and output operations per second) value, for example, if a machine can bear 1000 disk IOPS, when the disk utilization is 20%, it can be quantified as the used disk IOPS is 200.

[0013] 3. Adjust the performance reference value of each front-end group based on the quantified system performance data of each front-end group to obtain accurate external service capabilities of each front-end group (ie, historical resource usage data in the above-mentioned embodiment).

[0014] (IV) Bucket Profile Analysis

[0015] 1. Collect the original bucket read and write operation logs of each front-end group over the past two weeks and perform preliminary data processing on the bucket read and write operation logs, including but not limited to classifying the bucket read and write operation logs by operation type, removing invalid requests from the logs, and collecting statistics on the bucket read and write operation logs. After data processing, the bucket read and write operation logs of each front-end group over the past two weeks are obtained.

[0016] 2. Perform read / write analysis, behavior analysis, peak value analysis, etc. on the bucket read / write operation logs to obtain a bucket profile corresponding to the bucket read / write operation logs (i.e., the historical operation log data in the above embodiment of the specification). For example, a bucket profile may be:

[0017] "Time (timestamp) Number of read requests Number of write requests Number of delete requests Number of image processing times

[0018] 1701360000 100000 8000 2000 500

[0019] 1701363600 120000 9000 2500 550

[0020] 1701367200 80000 7000 1900 450 ”

[0021] 3. Based on the above method, determine the bucket profiles of each front-end group in multiple time periods, the external service capabilities of each front-end group corresponding to the multiple time periods, and the current external service capabilities of each front-end group.

[0022] 4. Based on the bucket profiles of each front-end group over multiple time periods and the external service capabilities of each front-end group corresponding to the multiple time periods, machine learning model building and model training are performed so that the trained machine learning model can understand the relationship between bucket profiles and external service capabilities. The trained machine learning model is the aforementioned resource prediction model.

[0023] 5. Based on the bucket portraits of multiple time periods grouped by each front-end, the time series prediction algorithm in the machine learning model is used to build and train the model. This allows the trained machine learning model to understand the relationship between the bucket portraits of the corresponding historical time periods and future time periods. The trained machine learning model is the aforementioned log prediction model.

[0024] 6. Input the bucket portrait of each front-end group within a preset time period in the past (for example, two weeks) into the log prediction model to obtain the bucket portrait within a preset time period in the future (for example, two weeks). Then, input the bucket portrait within the preset time period in the future (for example, two weeks) into the resource prediction model to obtain the external service capabilities required by each front-end group in the next two weeks.

[0025] (V) Decision Algorithm: The current external service capabilities of each front-end group and the external service capabilities required by each front-end group in the next two weeks are input into the decision algorithm. The decision algorithm provides a decision result (i.e., the above-mentioned decision strategy) based on the difference between the current external service capabilities of each front-end group and the external service capabilities required by each front-end group in the next two weeks, namely, whether each front-end group needs to be expanded / reduced, and the number of machines to be expanded / reduced (i.e., the number of nodes, where the machines are the above-mentioned data processing nodes).

[0026] (6) The operation and maintenance operation schedules resources for each front-end group based on the decision results given by the above decision algorithm. That is, if there is a front-end group that needs to be scaled down, that is, there is a high probability that the front-end group will be idle or have low resource usage in the next two weeks, then the front-end group that needs to be scaled down will expand its machines to the idle machine pool; if there is a front-end group that needs to be expanded, that is, there is a high probability that the front-end group will be resource-constrained in the next two weeks, then the machines that have been expanded in other front-end groups in the idle machine pool can be expanded to the front-end group that needs to be expanded; in addition, if the machines that have been expanded in other front-end groups in the idle machine pool are not sufficient to support the expansion to the front-end group that needs to be expanded, the machine delivery process will be triggered to ensure the normal expansion. The resource scheduling method provided in the embodiments of the present disclosure evaluates the external service capabilities of a front-end group and analyzes and predicts historical bucket operation logs using a machine learning model to determine the future resource capabilities required by the front-end group. This method then performs capacity expansion / contraction and resource rotation scheduling based on the front-end group's external service capabilities and future resource requirements. This improves resource utilization of the front-end group's data processing nodes, reduces resource waste, and enhances the user experience. Furthermore, the entire resource scheduling process is implemented through proactive, automatic, and intelligent scheduling by a computer, eliminating the need for human intervention. Timeliness is ensured, thereby guaranteeing the stability of cloud storage system services. Corresponding to the aforementioned method embodiments, the present disclosure also provides an embodiment of a resource scheduling device. Figure 4 is a schematic structural diagram of a resource scheduling device provided according to one embodiment of the present disclosure.As shown in Figure 4, the device includes: a historical data determination module 402, configured to determine historical operation log data and historical resource usage data generated by a data processing node set within a preset historical time period; a future operation log data prediction module 404, configured to predict the predicted operation log data generated by the data processing node set within a preset future time period based on the historical operation log data using a log prediction model; a future resource usage data prediction module 406, configured to predict the predicted resource usage data required by the data processing node set within the preset future time period based on the predicted operation log data using a resource prediction model; a target resource usage data determination module 408, configured to determine the target resource usage data of the data processing node set within the preset future time period based on the historical resource usage data; and a resource scheduling module 410, configured to perform resource scheduling on the data processing node set based on the predicted resource usage data and the target resource usage data. Optionally, the data processing node sets include at least two; the resource scheduling module 410 is further configured to: determine predicted resource usage data and target resource usage data for each data processing node set within the preset future time period; determine a decision strategy for each data processing node set based on the difference between the predicted resource usage data and the target resource usage data for each data processing node set within the preset future time period; and scale the data processing node sets up or down based on the decision strategy to implement resource scheduling for each data processing node set. Optionally, the resource scheduling module 410 is further configured to: determine a data processing node set to be expanded and / or a data processing node set to be scaled down based on the decision strategy; determine the number of nodes to be expanded in the data processing node set to be expanded and / or the number of nodes to be scaled down in the data processing node set to be scaled down; and scale the data processing node sets up or down based on the number of nodes to be expanded and / or the number of nodes to be scaled down. Optionally, the resource scheduling module 410 is further configured to: expand the data processing node sets according to the number of expanded nodes; or shrink the data processing node sets according to the number of shrunk nodes; or expand or shrink the data processing node sets according to the number of expanded nodes and the number of shrunk nodes.Optionally, the resource scheduling module 410 is further configured to: scale down the data processing nodes in the set of data processing nodes to be scaled down to the set of idle nodes based on the number of scaled-down nodes; and scale up the scaled-down idle data processing nodes in the set of idle nodes to the set of data processing nodes to be scaled up based on the number of scaled-up nodes, where the idle data processing nodes are any one or more data processing nodes in the set of data processing nodes to be scaled down. Optionally, the resource scheduling module 410 is further configured to: determine the number of idle data processing nodes in the set of idle nodes after scaling down; if it is determined that the number of scaled-up nodes is less than or equal to the number of idle nodes, determine a target idle data processing node from the idle data processing nodes based on the number of scaled-up nodes; and scale up the target idle data processing node to the set of data processing nodes to be scaled up. Optionally, the apparatus further includes a third-party expansion resource scheduling module configured to: if it is determined that the number of expansion nodes is greater than the number of idle nodes, determine the difference between the number of expansion nodes and the number of idle nodes; determine expansion data processing nodes from a third-party platform based on the difference; and expand the idle data processing nodes and the expansion data processing nodes to the set of data processing nodes to be expanded. Optionally, the historical data determination module 402 is further configured to: determine multiple storage spaces corresponding to the set of data processing nodes; determine a target storage space from the multiple storage spaces based on the number of accesses by the data processing node set to the multiple storage spaces; and obtain historical operation log data and historical resource usage data generated by the data processing node set accessing the target storage space within a preset historical time period. Optionally, the target resource usage data determination module 408 is further configured to: obtain simulated resource usage data generated by the data processing node set within the preset historical time period; and adjust the simulated resource usage data based on the historical resource usage data to obtain target resource usage data for the data processing node set within the preset future time period. Optionally, the apparatus further includes a log prediction model training module configured to: obtain sample historical operation log data generated within a plurality of preset historical time periods, and sample predicted operation log data generated within the preset future time period corresponding to each sample historical operation log data; and train an initial log prediction model based on the sample historical operation log data and the sample predicted operation log data until a training stop condition is met, thereby obtaining a log prediction model.Optionally, the apparatus further includes a resource prediction model training module configured to: obtain sample historical operation log data generated within a plurality of preset historical time periods, and sample historical resource usage data corresponding to each sample historical operation log data; and train an initial resource prediction model based on the sample historical operation log data and the sample historical resource usage data until a training stop condition is met, thereby obtaining a resource prediction model. Optionally, both the log prediction model and the resource prediction model are machine learning models. The above is a schematic diagram of a resource scheduling apparatus according to this embodiment. It should be noted that the technical solution of this resource scheduling apparatus and the technical solution of the resource scheduling method described above are based on the same concept. For details not described in detail in the technical solution of the resource scheduling apparatus, please refer to the description of the technical solution of the resource scheduling method described above. Referring to FIG. 5 , FIG. 5 shows a block diagram of a computing device 500 according to an embodiment of the present disclosure. Components of computing device 500 include, but are not limited to, a memory 510 and a processor 520. Processor 520 is connected to memory 510 via a bus 530, and database 550 is used to store data. The computing device 500 also includes an access device 540 that enables the computing device 500 to communicate via one or more networks 560. Examples of such networks include a public switched telephone network (PSTN), a local area network (LAN), a wide area network (WAN), a personal area network (PAN), or a combination of communication networks such as the Internet. The access device 540 may include one or more of any type of network interface (e.g., a network interface controller (NIC)) whether wired or wireless, such as an IEEE 802.11 wireless local area network (WLAN) wireless interface, a Worldwide Interoperability for Microwave Access (Wi-MAX) interface, an Ethernet interface, a universal serial bus (USB) interface, a cellular network interface, a Bluetooth interface, a near field communication (NFC) interface, and the like.In one embodiment of the present disclosure, the aforementioned components of computing device 500, as well as other components not shown in FIG. 5 , may also be connected to one another, for example, via a bus. It should be understood that the computing device block diagram shown in FIG. 5 is for illustrative purposes only and does not limit the scope of the present disclosure. Those skilled in the art may add or replace other components as needed. Computing device 500 may be any type of stationary or mobile computing device, including a mobile computer or mobile computing device (e.g., a tablet computer, personal digital assistant, laptop computer, notebook computer, netbook, etc.), a mobile phone (e.g., a smartphone), a wearable computing device (e.g., a smartwatch, smart glasses, etc.), or other types of mobile devices, or a stationary computing device such as a desktop computer or personal computer (PC). Computing device 500 may also be a mobile or stationary server. Processor 520 is configured to execute the following computer-executable instructions, which, when executed by the processor, implement the steps of the resource scheduling method described above. The above is a schematic diagram of a computing device according to this embodiment. It should be noted that the technical solution of this computing device is based on the same concept as the technical solution of the resource scheduling method described above. Any details not described in detail in the technical solution of the computing device can be found in the description of the technical solution of the resource scheduling method described above. An embodiment of the present disclosure also provides a computer-readable storage medium storing computer-executable instructions. When executed by a processor, these computer-executable instructions implement the steps of the resource scheduling method described above. The above is an exemplary embodiment of a computer-readable storage medium of this embodiment. It should be noted that the technical solution of this storage medium is based on the same concept as the technical solution of the resource scheduling method described above. Any details not described in detail in the technical solution of the storage medium can be found in the description of the technical solution of the resource scheduling method described above. An embodiment of the present disclosure also provides a computer program product comprising a computer program / instructions. When executed by a processor, these computer program / instructions implement the steps of the resource scheduling method described above. The above is an exemplary embodiment of a computer program product of this embodiment. It should be noted that the technical solution of this computer program product is based on the same concept as the technical solution of the resource scheduling method described above. Any details not described in detail in the technical solution of the computer program product can be found in the description of the technical solution of the resource scheduling method described above. Specific embodiments of the present disclosure have been described above. Other embodiments are within the scope of the following claims.In some cases, the actions or steps recited in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. Furthermore, the processes depicted in the drawings do not necessarily require the specific order or sequential order shown to achieve the desired results. In certain embodiments, multitasking and parallel processing are also possible or may be advantageous. The computer instructions include computer program code, which may be in source code form, object code form, executable files, or some intermediate form. The computer-readable medium may include any entity or device capable of carrying the computer program code, recording media, USB flash drives, removable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media. It should be noted that the content of the computer-readable medium may be appropriately increased or decreased based on the requirements of patent practice. For example, in some regions, according to patent practice, computer-readable media does not include electrical carrier signals or telecommunication signals. It should be noted that, for ease of description, the aforementioned method embodiments are described as a series of actions. However, those skilled in the art should understand that the embodiments of the present disclosure are not limited by the order of the actions described, as certain steps may be performed in a different order or simultaneously, depending on the embodiments of the present disclosure. Furthermore, those skilled in the art should also understand that the embodiments described in this specification are preferred embodiments, and the actions and modules described are not necessarily required for the embodiments of the present disclosure. In the above embodiments, the description of each embodiment has its own emphasis. For portions not described in detail in a particular embodiment, reference should be made to the relevant descriptions of other embodiments. The preferred embodiments disclosed above are merely intended to help illustrate the present disclosure. The optional embodiments do not describe all details in detail, nor do they limit the invention to the specific implementation methods described. Obviously, many modifications and variations are possible based on the content of the embodiments of the present disclosure. The present disclosure selects and describes these embodiments in detail to better explain the principles and practical applications of the embodiments of the present disclosure, thereby enabling those skilled in the art to better understand and utilize the present disclosure. The present disclosure is limited only by the claims and their full scope and equivalents.

Claims

Claims 1. A resource scheduling method, comprising: Determine the historical operation log data and historical resource usage data generated by the set of data processing nodes within a preset historical time period; Based on the historical operation log data, use a log prediction model to predict the predicted operation log data that the set of data processing nodes will generate within a preset future time period; based on the predicted operation log data, use a resource prediction model to predict the predicted resource usage data required by the set of data processing nodes within the preset future time period; based on the historical resource usage data, determine the target resource usage data of the set of data processing nodes within the preset future time period; based on the predicted resource usage data and the target resource usage data, perform resource scheduling on the set of data processing nodes.

2. The resource scheduling method according to claim 1, wherein the set of data processing nodes includes at least two; and the resource scheduling for the set of data processing nodes according to the predicted resource usage data and the target resource usage data includes: Determine the predicted resource usage data and target resource usage data of each set of data processing nodes within the preset future time period; Based on the difference between the predicted resource usage data and the target resource usage data of each set of data processing nodes within the preset future time period, determine the decision-making strategy for each set of data processing nodes; Based on the decision-making strategy, expand or contract each set of data processing nodes to achieve resource scheduling for each set of data processing nodes.

3. The resource scheduling method according to claim 2, wherein said expanding or contracting each data processing node set according to the decision strategy comprises: Based on the decision-making strategy, determine the set of data processing nodes to be expanded and / or the set of data processing nodes to be contracted; Determine the number of nodes to be expanded for the set of data processing nodes to be expanded and / or the number of nodes to be contracted for the set of data processing nodes to be contracted; based on the number of nodes to be expanded and / or the number of nodes to be contracted, expand or contract each set of data processing nodes.

4. The resource scheduling method according to claim 3, where expanding or shrinking each data processing node set according to the number of nodes to be expanded and / or the number of nodes to be shrunk includes: Based on the number of nodes to be expanded, expand each set of data processing nodes; Or based on the number of nodes to be contracted, contract each set of data processing nodes; Or based on the number of nodes to be expanded and the number of nodes to be contracted, expand or contract each set of data processing nodes. 22 5. The resource scheduling method according to claim 4, wherein the step of expanding or shrinking each data processing node set according to the number of nodes to be expanded and the number of nodes to be shrunk includes: Based on the number of nodes to be contracted, contract the data processing nodes in the set of data processing nodes to be contracted to the set of idle nodes; Based on the number of nodes to be expanded, expand the contracted idle data processing nodes in the set of idle nodes to the set of data processing nodes to be expanded, where the idle data processing nodes are any one or more data processing nodes in the set of data processing nodes to be contracted.

6. The resource scheduling method according to claim 5, wherein the step of expanding the scaled-down idle data processing nodes in the idle node set to the data processing node set to be expanded according to the number of nodes to be expanded includes: determining the number of idle nodes in the scaled-down idle data processing nodes in the idle node set; in the case where it is determined that the number of nodes to be expanded is less than or equal to the number of idle nodes, determining target idle data processing nodes from the idle data processing nodes according to the number of nodes to be expanded; and expanding the target idle data processing nodes to the data processing node set to be expanded.

7. The resource scheduling method according to claim 6, after determining the number of idle nodes of the idle data processing nodes after capacity reduction in the idle node set, further comprising: In the case where it is determined that the number of nodes to be expanded is greater than the number of idle nodes, determining the difference between the number of nodes to be expanded and the number of idle nodes; determining data processing nodes to be expanded from a third-party platform according to the difference; expanding the idle data processing nodes and the data processing nodes to be expanded to the data processing node set to be expanded.

8. The resource scheduling method according to any one of claims 1 to 7, wherein the determining the historical operation log data and historical resource usage data generated by the data processing node set within a preset historical time period includes: Determining a plurality of storage spaces corresponding to the data processing node set, and determining a target storage space from the plurality of storage spaces according to the access times of the data processing node set to the plurality of storage spaces; Obtaining historical operation log data and historical resource usage data generated by the data processing node set accessing the target storage space within a preset historical time period.

9. The resource scheduling method according to any one of claims 1 to 8, wherein the step of determining the target resource usage data of the data processing node set within the preset future time period according to the historical resource usage data includes: obtaining simulated resource usage data generated by the data processing node set within the preset historical time period; and adjusting the simulated resource usage data according to the historical resource usage data to obtain the target resource usage data of the data processing node set within the preset future time period.

10. The resource scheduling method according to any one of claims 1 to 9, wherein the step of predicting, according to the historical operation log data, the predicted... generated by the data processing node set within a preset future time period by using a log prediction model Before measuring the operation log data, it also includes: Obtaining sample historical operation log data generated within a plurality of the preset historical time periods and sample predicted operation log data generated within the preset future time period corresponding to each sample historical operation log data; training an initial log prediction model according to the sample historical operation log data and the sample predicted operation log data until a training stop condition is reached to obtain a log prediction model, wherein the log prediction model is a machine learning model.

11. The resource scheduling method according to any one of claims 1 to 10, before predicting the predicted resource usage data required by the data processing node set within the preset future time period by using the resource prediction model based on the predicted operation log data, further includes: Obtain the sample historical operation log data generated within multiple said preset historical time periods, as well as the sample historical resource usage data corresponding to each sample historical operation log data; train an initial resource prediction model according to the sample historical operation log data and the sample historical resource usage data until the training stop condition is reached to obtain a resource prediction model, wherein the resource prediction model is a machine learning model.

12. A resource scheduling device, comprising: A historical data determination module, configured to determine the historical operation log data and the historical resource usage data generated by a set of data processing nodes within a preset historical time period; A future operation log data prediction module, configured to predict the predicted operation log data generated by the set of data processing nodes within a preset future time period by using a log prediction model according to the historical operation log data; a future resource usage data prediction module, configured to predict the predicted resource usage data required by the set of data processing nodes within the preset future time period by using a resource prediction model according to the predicted operation log data; a target resource usage data determination module, configured to determine the target resource usage data of the set of data processing nodes within the preset future time period according to the historical resource usage data; A resource scheduling module, configured to perform resource scheduling on the set of data processing nodes according to the predicted resource usage data and the target resource usage data.

13. A computing device, comprising: A memory and a processor; The memory is used to store computer-executable instructions, and the processor is used to execute the computer-executable instructions. When the computer-executable instructions are executed by the processor, the steps of the resource scheduling method according to any one of claims 1 to 11 are implemented.

14. A computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, the steps of the resource scheduling method according to any one of claims 1 to 11 are implemented.

15. A computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the resource scheduling method according to any one of claims 1 to 11 are implemented.

Images