A method and system for scheduling and allocating business data

Abstract

The present invention discloses a business data scheduling and allocation method and system, which relates to the field of scheduling and allocation optimization technology, and solves the technical problem that the unified resource scheduling and allocation method has unreasonable resource allocation and affects the overall data processing efficiency. The present invention classifies the business data to be scheduled to obtain different types of data, and classifies the server nodes that process the data. When scheduling and allocating the data, the data is allocated and processed according to the load conditions of different nodes and the completion conditions of node task processing. On the one hand, the scheduling and allocation can be optimized in combination with the actual load conditions of the nodes to avoid data processing abnormalities caused by excessive load on a single node. On the other hand, the overall resource scheduling utilization rate is improved by analyzing the completion conditions and performing comprehensive scheduling in combination with the actual conditions of high-load nodes.

Description

Technical Field

[0001] The present invention relates to the technical field of scheduling and allocation optimization, and in particular to a method and system for scheduling and allocation of business data. Background Art

[0002] Scheduling optimization methods have wide application value in the fields of industry, economy, etc. The main purpose of real-time load scheduling under multiple electricity prices is to dispatch the load to the data center with low electricity price for cloud service through the front-end server under the premise of meeting SLAs, so as to minimize the electricity cost of the cloud provider.

[0003] According to publication number CN102752395B, an online scheduling method for real-time business distribution in a distributed data center is disclosed. The method can ensure that in the case of transmission delay, the feedback result of the specific distribution of the load flow path is used, and then the target is optimized by using a heuristic branch and bound method. The problem of parameter coupling of the delay constraint model in the mixed integer linear programming can be effectively solved, and the method is fast, accurate and efficient; especially when the number of front-end servers and data centers increases significantly, it can better reflect the advantages of high efficiency and speed.

[0004] However, some existing scheduling and allocation systems adopt a unified allocation method when scheduling tasks due to the diversity of data. Such an allocation method will lead to unreasonable resource scheduling and further affect the overall data processing efficiency. Summary of the invention

[0005] In view of the deficiencies in the prior art, the present invention provides a business data scheduling and allocation method and system, which solves the problem that in a unified resource scheduling and allocation method, unreasonable resource allocation exists, which affects the overall data processing efficiency.

[0006] To achieve the above objectives, the present invention is implemented through the following technical solutions: a service data scheduling and allocation method, comprising the following steps:

[0007] S101: Acquire all the to-be-adjusted business data and data processing nodes, and classify the to-be-adjusted business data and data processing nodes to obtain classification results, where the classification results include classification results of the to-be-adjusted business data and classification results of the data processing nodes;

[0008] S102: analyzing the classification results of the service data to be scheduled in the obtained classification results, and combining the classified service data to be scheduled according to data relevance to obtain a combined processing package;

[0009] S103: Distribute the service data to be called based on the data processing node classification results in the classification results, and select the node to be distributed by calculating the comprehensive value of the low-load node and combining the processing completion status of the corresponding node;

[0010] S104: performing allocation processing on the selected nodes to be allocated, performing scheduling with different priorities according to the classification type of the service data to be scheduled, and performing secondary scheduling allocation in combination with the processing tasks of the high-load nodes, and generating corresponding task scheduling information at the same time.

[0011] As a further solution of the present invention: the specific method of classifying the service data to be called and the data processing nodes in step S101 is:

[0012] Obtain all pending business data, extract data features from the pending business data, and classify the pending business data into regular business data and important business data based on the data features;

[0013] Obtain all data processing nodes, denoted as i, where i=1, 2, ..., j, where j represents the number of data processing nodes, and obtain the resource load corresponding to the data processing node i, denoted as Ki, and compare the obtained resource load Ki with the preset value Ky;

[0014] When the resource load Ki is greater than a preset value Ky, the corresponding data processing node is classified as a high-load node, otherwise the corresponding data processing node is classified as a low-load node.

[0015] As a further solution of the present invention: In step S102, the specific method of combining the classified service data to be scheduled according to the data association to obtain the combined processing package is:

[0016] Then, the classified business data to be scheduled is obtained, specifically, conventional business data and important business data. At the same time, the conventional business data and the important business data are analyzed for relevance based on the data mining algorithm, and the business data with relevance are combined to obtain a combined processing package;

[0017] The data capacity corresponding to the conventional business data and the important business data in the combined processing package is obtained, and the data type of the combined processing package is determined based on the type data with a larger data capacity, while the classified data without data correlation is not processed.

[0018] As a further solution of the present invention: the specific method of calculating the comprehensive value of the low-load node in step S103 is:

[0019] Obtain all low-load nodes in the data processing node classification results, and label the low-load nodes as a, where a=1, 2, ..., b, where b represents the number of low-load nodes, and obtain the processing speed Va, processing capacity La, and network bandwidth Ka corresponding to the low-load node a;

[0020] Substitute the parameters corresponding to the low-load node a into the formula The comprehensive value Qa corresponding to the low-load node a is calculated, where δ is a system built-in parameter, and the low-load nodes a are sorted from large to small according to the calculated comprehensive value Qa.

[0021] As a further solution of the present invention: the specific method of selecting the node to be allocated in step S103 in combination with the processing completion status of the node is:

[0022] According to the order of the comprehensive value Qa, the analysis is performed from large to small, the periodic processing volume corresponding to the low-load node a is obtained, and the real-time processing volume corresponding to the low-load node a is obtained, and the task completion ratio corresponding to the low-load node a is calculated, and then the task completion ratio corresponding to the low-load node a is compared with the threshold;

[0023] When the task completion ratio is greater than the threshold, the corresponding low-load node is marked as a node to be allocated, otherwise the low-load node is not processed.

[0024] As a further solution of the present invention: the specific method of performing scheduling with different priorities according to the classification type of the service data to be scheduled in step S104 is:

[0025] Obtain all combined processing packages, regular business data and important business data, and schedule and allocate them according to the priority of different business data. Obtain all nodes to be allocated and the idle processing capacity corresponding to the nodes to be allocated. At the same time, calculate and allocate the business data to be scheduled according to the idle processing capacity, and generate task scheduling information.

[0026] As a further solution of the present invention: the specific method of performing secondary scheduling allocation in step S104 in combination with the processing tasks of the high-load node is:

[0027] The remaining nodes to be allocated are obtained and recorded as secondary nodes to be allocated n, where n=1, 2, ..., m, where m represents the number of secondary nodes to be allocated. All high-load nodes are obtained, and the unprocessed important business data capacity in the high-load nodes is obtained, and the unprocessed important business data is sorted from large to small according to the data capacity of the unprocessed important business data;

[0028] Then, the idle processing capacity L1n and the corresponding comprehensive value Qn corresponding to the secondary node to be assigned n are obtained, and the sum of the two is calculated at the same time. The secondary node to be assigned with the largest sum is selected as the standard processing node to generate task scheduling information.

[0029] A business data scheduling and distribution system, comprising: a scheduling data information collection unit, a scheduling classification processing unit, a task scheduling and distribution unit and a scheduling information output unit;

[0030] A scheduling data information collection unit, which is used to transmit the acquired service data to be scheduled and the data processing node to the scheduling classification processing unit;

[0031] A scheduling classification processing unit is used to classify the acquired business data to be scheduled and the data processing nodes, obtain regular business data and important business data by classifying the business data to be scheduled, classify the data processing nodes into high-load nodes and low-load nodes, generate classification results, and transmit the classification results to the task scheduling allocation unit;

[0032] A task scheduling and allocation unit, which is used to perform scheduling and allocation according to the obtained classification results, process the classified business data to be adjusted according to the data correlation to obtain a combined processing package, then calculate the comprehensive value of the low-load node, and screen the low-load nodes according to the comprehensive value and the completion status of the node task processing to obtain the nodes to be allocated, and at the same time, schedule according to the priority according to the classification type of the business data to be adjusted to generate task scheduling information, and perform secondary scheduling and allocation on the important business data corresponding to the high-load node to generate task scheduling information, and at the same time transmit the task scheduling information to the scheduling information output unit;

[0033] The scheduling information output unit is used to display the generated task scheduling information to the corresponding operator.

[0034] The present invention provides a method and system for scheduling and allocating business data. Compared with the prior art, the present invention has the following beneficial effects:

[0035] The present invention classifies the service data to be scheduled to obtain different types of data, and classifies the server nodes that process the data. When scheduling and allocating the data, the data is allocated and processed according to the load conditions of different nodes and the completion status of node task processing. On the one hand, it can optimize the scheduling and allocation in combination with the actual load conditions of the nodes to avoid data processing abnormalities caused by excessive load on a single node. On the other hand, it analyzes the completion status and performs comprehensive scheduling in combination with the actual conditions of high-load nodes, thereby improving the overall resource scheduling utilization rate. BRIEF DESCRIPTION OF THE DRAWINGS

[0036] Figure 1 It is a flow chart of the steps of the present invention;

[0037] Figure 2 This is a block diagram of the system principle of the present invention. DETAILED DESCRIPTION

[0038] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0039] For example, see Figure 1 , the present application provides a method for scheduling and allocating business data, Figure 1 It is a flowchart of an example of the business data scheduling and allocation method of the present invention.

[0040] Next, in step S101, all the business data to be called and the data processing nodes are obtained, and the business data to be called and the data processing nodes are classified to obtain classification results, and the classification results here include the classification results of the business data to be called and the classification results of the data processing nodes.

[0041] First, all the business data to be adjusted are obtained, and data features are extracted for the business data to be adjusted. At the same time, the business data to be adjusted is classified according to the data features to obtain regular business data and important business data. Data analysis algorithms and tools (such as data mining algorithms, statistical analysis methods) are used to conduct in-depth analysis on the cleaned data to extract key business features. These features may include the periodicity of business volume, the distribution of business types, the timeliness requirements of data, etc. Based on the extracted features, a classification and labeling system for business data is established to further classify the business data to be adjusted.

[0042] For example, in an e-commerce business scenario, daily browsing data of ordinary goods is classified as regular business data, while transaction data involving high-value goods is classified as important business data.

[0043] Then, all data processing nodes are classified and labeled as i, where i=1, 2, ..., j, where j represents the number of data processing nodes, and specific data processing nodes include servers, computing clusters, etc. At the same time, the resource load corresponding to the data processing node i is obtained and recorded as Ki, and the resource load here represents the network bandwidth occupancy rate corresponding to the corresponding data processing node, and the obtained resource load Ki is compared with the preset value Ky, and the specific value of the preset value here is set by the operator;

[0044] When the resource load Ki is greater than the preset value Ky, including the case where it is equal to the preset value Ky, the corresponding data processing node is classified as a high-load node. Conversely, when the resource load Ki is less than the preset value Ky, the corresponding data processing node is classified as a low-load node.

[0045] In a specific example, there are 5 data processing nodes, namely 3 servers and 2 computing clusters. Assume that the operator sets Ky=60% (that is, when the network bandwidth occupancy rate reaches 60% as a judgment limit) based on experience and network performance requirements. For one of the servers (labeled as i=1), if its network bandwidth occupancy rate K1=70%, since K1 is greater than Ky, then this server is classified as a high-load node. For another computing cluster (labeled as i=4), if its network bandwidth occupancy rate K4=50%, since K4 is less than Ky, the computing cluster is classified as a low-load node.

[0046] Next, in step S102, the classification results of the service data to be scheduled in the obtained classification results are analyzed, and the classified service data to be scheduled are combined and processed according to data relevance to obtain a combined processing package.

[0047] Then, the classified business data to be scheduled is obtained, specifically, conventional business data and important business data. At the same time, a correlation analysis is performed on the conventional business data and the important business data based on a data mining algorithm, such as an April algorithm, an FP-Growth algorithm, etc., and the business data with correlation is combined to obtain a combined processing package. The data types included in the combined processing package here are conventional business data and important business data. At the same time, the composition method may be a single conventional business data, a single important business data, and a mixture of conventional business data and important business data. Then, the data capacity corresponding to the conventional business data and the important business data in the combined processing package is obtained, and the type of data with a larger data capacity is selected to determine the data type of the combined processing package. Classified data without data correlation is not processed.

[0048] In a specific example, for example, in an e-commerce business scenario, in a combined processing package, regular business data is the record of users browsing a certain type of goods, with a capacity of 50MB, and important business data is the user's purchase record of this type of goods, with a capacity of 30MB. Since 50MB (regular business data capacity) is greater than 30MB (important business data capacity), the data type of the combined processing package is determined to be the regular business data type.

[0049] Next, in step S103, the business data to be called is allocated and processed based on the data processing node classification results in the classification results, and the nodes to be allocated are selected by calculating the comprehensive values ​​of the low-load nodes and combining the processing completion status of the corresponding nodes.

[0050] Obtain all low-load nodes in the data processing node classification results, and label the low-load nodes as a, and a=1, 2, ..., b, where b represents the number of low-load nodes, and obtain the processing speed Va corresponding to the low-load node a (the processing speed represents the amount of computing tasks or data processing that can be completed in a unit of time. If node a can complete 1000 mathematical operations within 1 second, then its processing speed Va can be expressed as 1000 operations / second; in the data processing scenario, if node a can process 10MB of data per second, then Va is 10MB / second) and processing capacity La (the processing capacity La generally refers to the upper limit of the data scale or task scale that node a can process simultaneously. It reflects the capacity of node a in terms of processing power. For a database server node a, its processing capacity La may be able to process 1000 concurrent database query requests at the same time. ; or for an image rendering node, La can mean that it can handle the rendering tasks of 50 high-resolution images at the same time) and network bandwidth Ka (network bandwidth Ka refers to the amount of data that can be transmitted per unit time when node a transmits data with other nodes or networks. It determines the data transmission capacity of node a in network communication. It is usually expressed in bits per second (bps) or bytes per second (Bps). For example, common ones are 10Mbps, 100Mbps, 1Gbps, etc. If node a can receive 100Mbps of data from the network within 1 second, then its network bandwidth Ka is 100Mbps; or in the file upload scenario, if node a can upload files to the remote server at a speed of 5MB / second, then Ka is the effective bandwidth of this upload process. 5MB / second), then substitute the parameters corresponding to the low-load node a into the formula The comprehensive value Qa corresponding to the low-load node a is calculated, where δ is a system built-in parameter, and the specific value is set by the operator. At the same time, the low-load nodes a are sorted from large to small according to the calculated comprehensive value Qa;

[0051] Then, the analysis is performed in descending order according to the sorting order of the comprehensive value Qa, and the periodic processing volume corresponding to the low-load node a is obtained. The periodic processing volume here is calculated based on past data, and the corresponding periodic processing volume is expressed as an average. At the same time, the real-time processing volume corresponding to the low-load node a is obtained, and the real-time processing volume here represents the number of tasks that the node has completed processing, and the task completion ratio corresponding to the low-load node a is calculated. The specific task completion ratio is obtained by dividing the real-time processing volume by the periodic processing volume. Then, the task completion ratio corresponding to the low-load node a is compared with the threshold, and the specific value of the threshold is set by the operator. When the task completion ratio is greater than the threshold, the corresponding low-load node is marked as a node to be allocated. Conversely, when the task completion ratio is less than the threshold, the low-load node is not processed.

[0052] In a specific example, for example, in a data processing center, there is a low-load node a. By analyzing the data of the past month, it is found that it can process an average of 1,000 data processing tasks per day (this is the periodic processing volume), and at the current moment, it has completed 600 tasks (this is the real-time processing volume). Then the task completion percentage is 600÷1000=0.6, and then the task completion percentage of the low-load node a is compared with the threshold (the threshold is set by the operator according to actual conditions).

[0053] Assume that the operator sets the threshold value to 0.5. Since 0.6>0.5, the low-load node is marked as a node to be allocated. If the calculated task completion ratio is less than 0.5, the low-load node is not processed.

[0054] Next, in step S104, the selected nodes to be allocated are allocated, different priorities are scheduled according to the classification type of the service data to be scheduled, and secondary scheduling is performed in combination with the processing tasks of the high-load nodes, and corresponding task scheduling information is generated.

[0055] Obtain all combined processing packages, regular business data and important business data, and schedule and allocate them according to the priority of different business data. The priority here is to schedule and allocate important business data first. For combined processing packages, if the combined processing packages are characterized as important business data or regular business data, the corresponding priority is higher than that of regular business data, but lower than that of important business data. The specific scheduling and allocation method is: obtain all nodes to be allocated, and obtain the idle processing capacity corresponding to the nodes to be allocated. The idle processing capacity here represents the data capacity that the corresponding node can continue to process. At the same time, the business data to be called is calculated and allocated according to the idle processing capacity, and task scheduling information is generated. The specific calculation and allocation method is: first allocate the important business data in the business data to be called, obtain the data capacity of the important business data, determine the number that can continue to be processed according to the idle processing capacity, and further generate allocation information. After the allocation of important business data is completed, the subsequent combined processing packages and regular business data are allocated. All business data to be called are allocated in this way;

[0056] In a specific example, suppose there are three nodes A, B, and C to be allocated, the idle processing capacity of node A is 50GB, the idle processing capacity of node B is 30GB, and the idle processing capacity of node C is 40GB. Now there are 60GB of important business data, 40GB of combined processing packages (qualitatively with a priority higher than regular business data and lower than important business data), and 30GB of regular business data that need to be allocated. First, the important business data is allocated, 50GB of important business data is allocated to node A, and 10GB is allocated to node B. Then the combined processing packages are allocated, 30GB is allocated to node B, 10GB is allocated to node C, and finally the regular business data is allocated, 30GB is allocated to node C. In this way, the allocation of all business data is completed and the corresponding task scheduling information is generated.

[0057] Then, the remaining nodes to be allocated are obtained and recorded as secondary nodes to be allocated n, and n = 1, 2, ..., m, where m represents the number of secondary nodes to be allocated. All high-load nodes are obtained, and the unprocessed important business data capacity in the high-load nodes is obtained, and they are sorted from large to small according to the data capacity of the unprocessed important business data. Then, the idle processing capacity L1n and the corresponding comprehensive value Qn corresponding to the secondary node to be allocated n are obtained, and the sum of the two is calculated at the same time. The secondary node to be allocated with the largest sum is selected as the standard processing node, and the task scheduling information is generated. The analysis is carried out in this way.

[0058] For example 2, please refer to Figure 2A business data scheduling and allocation system, the system includes: a scheduling data information acquisition unit, a scheduling classification processing unit, a task scheduling allocation unit, and a scheduling information output unit, and the above-mentioned functional units are unidirectionally electrically connected.

[0059] A scheduling data information collection unit, which is used to transmit the acquired service data to be scheduled and the data processing node to the scheduling classification processing unit;

[0060] A scheduling classification processing unit is used to classify the acquired business data to be scheduled and data processing nodes, and obtain regular business data and important business data by classifying the business data to be scheduled, and obtain high-load nodes and low-load nodes by classifying the data processing nodes, and generate classification results. At the same time, the classification results are transmitted to the task scheduling allocation unit, and the processing method here is the same as the processing method of step S101 in Example 1.

[0061] A task scheduling and allocation unit is used to perform scheduling and allocation according to the obtained classification results, and to process the classified business data to be called according to the data correlation to obtain a combined processing package, and the processing method here is the same as the processing method of step S102 in the first embodiment, and then calculate the comprehensive value of the low-load node, and screen the low-load node according to the comprehensive value and the completion status of the node task processing to obtain the node to be allocated, and the processing method here is the same as the processing method of step S013 in the first embodiment, and at the same time, according to the classification type of the business data to be called, schedule according to priority to generate task scheduling information, and perform secondary scheduling and allocation on the important business data corresponding to the high-load node to generate task scheduling information, and at the same time transmit the task scheduling information to the scheduling information output unit, and the specific method of generating the task scheduling information here is the same as the processing method of step S104 in the first embodiment.

[0062] The scheduling information output unit is used to display the generated task scheduling information to the corresponding operator.

[0063] Some of the data in the above formulas are dimensionless and numerically calculated. Meanwhile, the contents not described in detail in this specification belong to the prior art known to those skilled in the art.

[0064] The above embodiments are only used to illustrate the technical method of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical method of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical method of the present invention.

Claims

1. A method for scheduling and allocating service data, characterized in that: The following steps are involved: S101: Acquire all the to-be-adjusted business data and data processing nodes, and classify the to-be-adjusted business data and data processing nodes to obtain classification results, where the classification results include classification results of the to-be-adjusted business data and classification results of the data processing nodes; S102: analyzing the classification results of the service data to be scheduled in the obtained classification results, combining the classified service data to be scheduled according to data relevance to obtain a combined processing package, and determining the data type of the combined processing package based on the type data with a larger data capacity; S103: Based on the data processing node classification results in the classification results, the to-be-allocated business data is allocated and processed, and the nodes to be allocated are selected by calculating the comprehensive value of the low-load nodes and combining the processing completion status of the corresponding nodes, and all the data processing nodes are obtained and recorded as i, and i=1, 2, ..., j, where j represents the number of data processing nodes, and the resource load corresponding to the data processing node i is obtained and recorded as Ki, and the obtained resource load Ki is compared with the preset value Ky; When the resource load Ki is greater than the preset value Ky, the corresponding data processing node is classified as a high-load node, otherwise the corresponding data processing node is classified as a low-load node; The low-load nodes are labeled as a, and a=1, 2, ..., b, where b represents the number of low-load nodes; Calculation formula The comprehensive value Qa corresponding to the low-load node a is calculated, where is the system built-in parameter, Va, La and Ka are the processing speed, processing capacity and network bandwidth corresponding to the load node a respectively, and is to sort the low-load nodes a from large to small according to the calculated comprehensive value Qa; Analyze from large to small according to the sorting order of the comprehensive value Qa, obtain the periodic processing volume and real-time processing volume corresponding to the low-load node a, and calculate the task completion ratio corresponding to the low-load node a, and then compare the task completion ratio corresponding to the low-load node a with the threshold; When the task completion ratio is greater than the threshold, the corresponding low-load node is marked as a node to be allocated; S104: performing allocation processing on the selected nodes to be allocated, performing scheduling with different priorities according to the classification type of the service data to be allocated, and performing secondary scheduling allocation in combination with the processing tasks of the high-load nodes, and generating corresponding task scheduling information at the same time, recording the remaining nodes to be allocated as secondary nodes to be allocated n, and n=1, 2, ..., m, where m represents the number of secondary nodes to be allocated, obtaining the unprocessed important service data capacity in the high-load nodes, and sorting the unprocessed important service data from large to small according to the data capacity; Then, the idle processing capacity L1n and the corresponding comprehensive value Qn corresponding to the secondary node to be assigned n are obtained, and the sum of the two is calculated at the same time. The secondary node to be assigned with the largest sum is selected as the standard processing node to generate task scheduling information.

2. A method for scheduling and allocating service data according to claim 1, characterized in that: The specific method of performing scheduling with different priorities according to the classification type of the service data to be scheduled in step S104 is: Obtain all combined processing packages, regular business data and important business data, and schedule and allocate them according to the priority of different business data. Obtain all nodes to be allocated and the idle processing capacity corresponding to the nodes to be allocated. At the same time, calculate and allocate the business data to be scheduled according to the idle processing capacity, and generate task scheduling information.

3. A service data scheduling and allocation system, used to execute a service data scheduling and allocation method according to any one of claims 1 to 2, characterized in that: include: Scheduling data information collection unit, scheduling classification processing unit, task scheduling allocation unit and scheduling information output unit; The dispatching data information collection unit is used to transmit the acquired service data to be dispatched and the data processing node to the dispatching classification processing unit; The scheduling classification processing unit is used to classify the acquired business data to be scheduled and the data processing nodes, obtain the conventional business data and the important business data by classifying the business data to be scheduled, classify the data processing nodes into the high-load nodes and the low-load nodes, generate the classification results, and transmit the classification results to the task scheduling allocation unit; The task scheduling and allocation unit is used to perform scheduling and allocation according to the obtained classification results, process the classified business data to be adjusted according to the data correlation to obtain a combined processing package, then calculate the comprehensive value of the low-load node, and screen the low-load nodes according to the comprehensive value and the completion status of the node task processing to obtain the nodes to be allocated, and at the same time, schedule according to the priority according to the classification type of the business data to be adjusted to generate task scheduling information, and perform secondary scheduling and allocation on the important business data corresponding to the high-load node to generate task scheduling information, and transmit the task scheduling information to the scheduling information output unit; The scheduling information output unit is used to display the generated task scheduling information to the corresponding operator.

Images