Scheduling method and system based on cluster

Abstract

The invention discloses a cluster-based scheduling method and system. The method comprises the following steps: S1, analyzing batch service requests from the same address and the total data writing amount of the batch service requests; s2, judging whether the cluster of the area to which the batch service requests belong meets a write-in condition, if so, selecting the cluster of the area to which the batch service requests belong as a write-in cluster of the batch service requests, and if not, executing the step S3; and S3, sorting according to the candidate clusters accessed by the batch of service requests, and taking the candidate cluster which is optimally sorted and meets the write-in condition as a write-in cluster. When the dynamic change of the service request is not matched with the capacity of the local cluster, the address of the service request, the current to-be-recycled space of each cluster, the residual available space, the load and the recycling speed are comprehensively considered, and the service request is sent to the optimal cluster, so that the resources of each cluster are efficiently and fully utilized; and cluster resources can be adjusted in time under the continuous condition of not interrupting the service.

Description

Technical Field

[0001] The present application relates to the field of network storage technology, and in particular to a cluster-based scheduling method and system. Background Art

[0002] The most important technical directions in the information age are mainly based on the continuous development and progress of storage, computing and transmission. With the popularization of technology, a huge amount of data files are generated on the Internet. For massive data files, in the field of storage, currently distributed storage solutions are basically adopted to meet the needs of massive storage and access. Distributed storage is a technology that stores data in multiple physical nodes in a dispersed manner, aiming to improve the availability, reliability and scalability of data.

[0003] Cluster-based distributed storage systems are usually composed of multiple servers (nodes) that collaborate with each other in the network to achieve unified data storage management. When existing distributed storage is deployed in multiple clusters or multiple regions, the total capacity, available space and other status of each cluster are different. In most business scenarios, all requests are processed locally without a centralized request scheduling system. If business requests to each storage cluster are not multi-scheduled and are all requested to the local region or cluster, there will be a waste of space resources or no space available while the storage space of other clusters is idle, resulting in the inability to fully utilize the storage space of each cluster. Summary of the invention

[0004] In order to solve the technical problems existing in the prior art, the present application proposes a cluster-based scheduling method, comprising the following steps:

[0005] S1: Analyze batch business requests from the same address and their total data write volume;

[0006] S2: Determine whether the cluster in the region to which the batch service request belongs meets the write condition. If yes, select the cluster in the region as the write cluster for the batch request. If no, execute step S3.

[0007] S3: Sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing conditions as the writing cluster.

[0008] Furthermore, step S4 is included: if each cluster does not meet the write condition, the remaining space of each cluster and the total amount of space to be reclaimed are sorted, and the write cluster is taken as the write cluster according to the following write formula condition:

[0009] Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx%

[0010] Where: Hx is the remaining available space, Rx is the recycling speed, Tx is the estimated writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number;

[0011] If all clusters do not meet the conditions, the batch request will be rejected or the request will be cached, and the cluster will be reselected for writing after the preset time interval.

[0012] Furthermore, a batch service request is N consecutive service requests.

[0013] Furthermore, write the condition:

[0014] Hx-Dx>Cx*(1-Mx%), and the load is lower than Lx%.

[0015] Furthermore, the sorting in step S3 is performed according to the latency of the batch of service requests accessing the candidate cluster.

[0016] Furthermore, the sorting in step S3 is performed according to the remaining available space of the candidate clusters accessed by the batch of business requests.

[0017] The present application also provides a cluster-based scheduling system for implementing the cluster-based scheduling method, comprising an analysis unit, a judgment unit and a sorting unit, wherein:

[0018] The analysis unit is used to analyze batch service requests from the same address and their total data writing amount;

[0019] The judging unit is used to judge whether the cluster of the area to which the batch service request belongs meets the writing condition. If so, the cluster of the area to which the batch service request belongs is selected as the writing cluster of the batch service request. If not, the cluster is processed according to the executing unit.

[0020] The sorting unit is used to sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing condition as the writing cluster.

[0021] Furthermore, an optimization unit is included, which is used to sort the remaining space of each cluster and the total amount of space to be reclaimed if each cluster does not meet the writing condition, and use the following writing conditions as the writing cluster:

[0022] Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx%

[0023] Where: Hx is the remaining available space, Rx is the recycling speed, Tx is the estimated writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number;

[0024] If all clusters do not meet the conditions, the batch request will be rejected or the request will be cached, and the cluster will be reselected for writing after the preset time interval.

[0025] The present invention also provides a storage medium, on which a computer program is stored, and when the computer program is run by a processor, the cluster-based scheduling method is executed.

[0026] The present invention also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to execute the cluster-based scheduling method.

[0027] In actual applications, the modules described in the method and system disclosed in the present application can be deployed in multiple modules on one target server, or each module can be independently deployed on different target servers. In particular, as needed, in order to provide more powerful computing and processing capabilities, the modules can also be deployed on cluster target servers as needed.

[0028] It can be seen that when the business request changes dynamically and does not match the local cluster capacity, the application comprehensively considers the address of the business request, the current space to be reclaimed in each cluster, the remaining available space, the load, and the recovery speed, and sends the business request to the best cluster, so as to efficiently and fully utilize the resources of each cluster, and can adjust the cluster resources in time without interrupting the continuity of the business.

[0029] In order to have a clearer and more comprehensive understanding of the present application, the specific implementation methods of the present application are described in detail below in conjunction with the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings based on these drawings without creative work.

[0031] Figure 1 For the embodiments of this application Figure 1 This is a schematic diagram of a multi-cluster scheduling structure in an embodiment of the present application.

[0032] Figure 2 A schematic diagram of a flow chart of a cluster-based scheduling method according to an embodiment of the present application. DETAILED DESCRIPTION

[0033] See also Figure 1, Figure 1 This is a schematic diagram of multi-cluster scheduling in an embodiment of the present application. In this embodiment, cluster a, cluster b, and cluster c are taken as examples. Each cluster has a cluster information collection module that collects storage information of the current cluster, including: available space, data deleted and waiting to be recycled space, remaining available space, load, etc. The scheduling center obtains relevant information of each cluster by reporting or pulling from the scheduling center.

[0034] This application proposes an improved cluster-based scheduling method that fully considers factors such as the current available space, space to be reclaimed, reclaim speed, data access effect, etc. in multiple clusters to achieve efficient and full utilization of the resources of each cluster. The solution includes the following steps:

[0035] S1: Analyze batch business requests from the same address and their total data write volume;

[0036] S2: Determine whether the cluster in the region to which the batch service request belongs meets the write condition. If yes, select the cluster in the region as the write cluster for the batch request. If no, execute step S3.

[0037] S3: Sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing conditions as the writing cluster.

[0038] See also Figure 2 , the following combination Figure 2 The flow chart shown illustrates the technical solution of the present application in detail.

[0039] S1: Analyze batch business requests from the same address and their total data write volume.

[0040] The batch of this application refers to business requests from the same address, such as address a, which is N consecutive business requests. Such multiple consecutive business requests are scheduled as batches, which can reduce the decision-making calculation cost and ensure that the data are stored in the same cluster. When part of the data is accessed, the cache system inside this cluster can load in batches to increase the access speed and reduce the delay of non-local access.

[0041] The total data write amount is calculated as d by taking the continuous N business requests as a batch, which further reduces the computational cost of word access.

[0042] S2: Determine whether the cluster in the region to which the batch service request belongs meets the write condition. If yes, select the cluster in the region as the write cluster for the batch request. If no, execute step S3.

[0043] In general, first select cluster a of the region corresponding to the batch request service initiated by address a. The total data write amount of the batch request is D. Then determine whether it meets the write condition. The formula of the write condition is as follows:

[0044] Hx-Dx>Cx*(1-Mx%), and the load is lower than Lx%.

[0045] In the calculation of this formula, it can be preset that the M% of each cluster can be different, usually 85% to 99%, and the load is lower than L%. In this way, it can be determined whether the threshold of the maximum storage capacity percentage of the cluster will be exceeded after storing the data of this batch of requests, and the load is low, which will not cause the risk of exceeding the cluster failure carrying capacity.

[0046] After judgment, if yes, cluster a is selected as the write cluster for this batch request.

[0047] If not, further judgment is required, namely step S3.

[0048] S3: Sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing conditions as the writing cluster.

[0049] As a preferred embodiment, the sorting of the present application is based on the latency of the batch of business requests accessing the candidate clusters. For example, for area a, after analysis in step S2, the threshold is exceeded and cluster a cannot be selected. However, the latency between cluster b and area a is lower and does not exceed the threshold, so cluster b is selected as the cluster selected for this batch of requests.

[0050] As an alternative preferred embodiment, the sorting of the present application can also be performed based on the remaining available space of the candidate clusters accessed by the batch of business requests. After the remaining available space of the candidate clusters is sorted from large to small, the cluster with the largest remaining available space is selected as the cluster for this batch of requests without considering the problem of long latency.

[0051] In addition, the present application also considers other scheduling situations for further optimization. The embodiment of the present application also includes step S4: if each cluster does not meet the write condition, further optimization selection is required. The present application sorts the remaining space of each cluster and the total amount of space to be reclaimed, and uses the following write conditions as the write cluster:

[0052] Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx%

[0053] Among them: Hx is the remaining available space, Rx is the recycling speed, Tx is the expected writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number.

[0054] If all clusters do not meet the conditions, this batch request will be rejected. Since the storage space of each cluster in this application is changing dynamically, you can also request caching, that is, cache the batch request, wait for the preset time interval, and then re-execute the above process of this application, select the appropriate cluster and then write.

[0055] It can be seen that when the business request changes dynamically and does not match the local cluster capacity, when the dispatch center receives the business request, the address of the business request, the current space to be reclaimed, the remaining available space, the load, and the recovery speed of each cluster are comprehensively considered, and the business request is sent to the best cluster, so as to achieve efficient and full use of the resources of each cluster, and the cluster resources can be adjusted in time without interrupting the continuity of the business, especially in the following scenarios to achieve better technical effects:

[0056] For example, some cluster services are not busy during a period of time, so there is a lot of available space. At this time, the space will be vacant for a long time and not fully utilized. However, it is difficult to reduce the capacity of the cluster and free up physical equipment in a short period of time. At this time, the scheduling system needs to schedule requests to fully utilize this space.

[0057] For example, there is a sudden surge in business requests in a certain area, but there is little available space. Due to non-technical external reasons such as procurement and processes, it is difficult to complete cluster expansion within a limited time. However, the business system cannot be suspended, so requests need to be dispatched to other clusters.

[0058] Especially for scenarios with frequent writing and deletion such as video surveillance, for most storage systems, it takes a certain amount of time to reclaim and release space. Therefore, deleted data that has not been completely recovered will occupy a part of the space. If factors such as the speed of space release are fully considered, the data storage space can be fully utilized.

[0059] Based on the above method embodiment, the present application also provides a cluster-based scheduling system for implementing the cluster-based scheduling method, which includes an analysis unit, a judgment unit, and a sorting unit, wherein:

[0060] The analysis unit is used to analyze batch service requests from the same address and their total data writing amount;

[0061] The judging unit is used to judge whether the cluster of the area to which the batch service request belongs meets the writing condition. If so, the cluster of the area to which the batch service request belongs is selected as the writing cluster of the batch service request. If not, the cluster is processed according to the executing unit.

[0062] The sorting unit is used to sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing condition as the writing cluster.

[0063] Furthermore, an optimization unit is included, which is used to sort the remaining space of each cluster and the total amount of space to be reclaimed if each cluster does not meet the writing condition, and use the following writing conditions as the writing cluster:

[0064] Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx%

[0065] Among them: Hx is the remaining available space, Rx is the recycling speed, Tx is the estimated writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number.

[0066] If all clusters do not meet the conditions, the batch request will be rejected or the request will be cached, and the cluster will be reselected for writing after the preset time interval.

[0067] An embodiment of the present application further provides a storage medium, on which a computer program is stored. When the computer program is executed by a processor, the cluster-based scheduling method as described above is executed.

[0068] An embodiment of the present application also provides an electronic device, comprising: a processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor communicates with the storage medium via the bus, and the processor executes the machine-readable instructions to perform any of the cluster-based scheduling methods described above.

[0069] It should be noted that a person skilled in the art can understand that all or part of the split steps in the various methods of the above embodiments can be completed by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and the storage medium can include but is not limited to: read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0070] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A cluster-based scheduling method, characterized in that: Includes steps: S1: Analyze batch business requests from the same address and their total data write volume; S2: Determine whether the cluster in the region to which the batch service request belongs meets the write condition. If yes, select the cluster in the region as the write cluster for the batch request. If no, execute step S3. S3: Sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing conditions as the writing cluster.

2. The cluster-based scheduling method according to claim 1, characterized in that: The process also includes step S4: if each cluster does not meet the write condition, the remaining space of each cluster and the total amount of space to be reclaimed are sorted, and the clusters are written according to the following write formula conditions: Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx% Where: Hx is the remaining available space, Rx is the recycling speed, Tx is the estimated writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number; If all clusters do not meet the conditions, the batch request will be rejected or the request will be cached, and the cluster will be reselected for writing after the preset time interval.

3. The cluster-based scheduling method according to claim 1, characterized in that: A batch service request is N consecutive service requests.

4. The cluster-based scheduling method according to claim 1, characterized in that: Write conditions: Hx-D>Cx*(1-Mx%), and the load is lower than Lx%.

5. The cluster-based scheduling method according to claim 1, characterized in that ,The sorting in step S3 is based on the latency of this batch of ,business requests to access the candidate clusters.

6. The cluster-based scheduling method according to claim 1, characterized in that ,The sorting in step S3 is to sort the batch of ,business requests based on the remaining available space of the candidate clusters that ,are accessed.

7. A cluster-based scheduling system, used to implement the cluster-based scheduling method according to any one of claims 1 to 6, characterized in that: It includes an analysis unit, a judgment unit and a sorting unit, wherein: The analysis unit is used to analyze batch service requests from the same address and their total data writing amount; The judging unit is used to judge whether the cluster of the area to which the batch service request belongs meets the writing condition. If so, the cluster of the area to which the batch service request belongs is selected as the writing cluster of the batch service request. If not, the cluster is processed according to the executing unit. The sorting unit is used to sort the candidate clusters accessed by the batch of business requests, and use the candidate cluster with the best sorting and meeting the writing condition as the writing cluster.

8. The cluster-based scheduling system according to claim 7, characterized in that: It also includes an optimization unit for sorting the remaining space of each cluster and the total amount of space to be reclaimed if each cluster does not meet the write condition, and taking the write cluster as the write cluster according to the following write condition: Hx+Rx*Tx-D>Cx*(1-Mx%), and the load is lower than Lx% Where: Hx is the remaining available space, Rx is the recycling speed, Tx is the estimated writing time, D is the total data written in this batch, Cx is the total capacity, Mx is the adjustment parameter, Lx is the load, and x is the cluster number; If all clusters do not meet the conditions, the batch request will be rejected or the request will be cached, and the cluster will be reselected for writing after the preset time interval.

9. A storage medium, characterized in that: The storage medium stores a computer program, and when the computer program is executed by the processor, the cluster-based scheduling method according to any one of claims 1 to 6 is executed.

10. An electronic device, comprising: A processor, a storage medium and a bus, wherein the storage medium stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor The processor communicates with the storage medium via a bus, and the processor executes the machine-readable instructions. To execute a cluster-based scheduling method as described in any one of claims 1 to 6.

Images