Data scheduling method, device and electronic equipment

Abstract

The embodiments of the present application relate to the field of database technology, and disclose a data scheduling method, device, and electronic device, the method comprising: obtaining scheduling parameters of a storage node in a first scheduling period; wherein the scheduling parameters are determined according to predicted load data of the first scheduling period; the predicted load data is predicted according to historical data; the historical data includes data of historical scheduling periods before the first scheduling period; and the data of the storage node is scheduled according to the scheduling parameters. The embodiments of the present application solve the problem in the prior art that, in the process of database scheduling data, the degree of dependence on the accuracy of the model itself and the amount of trained data is high.

Description

Technical Field

[0001] The present application relates to the field of database technology, and in particular to a data scheduling method, device and electronic device. Background Art

[0002] In the database field, it is often necessary to schedule data between the storage nodes (stores) of the database, such as distributed databases. In order to improve the efficiency of data scheduling, in related technologies, the load of the storage node in the subsequent time is usually predicted based on the prediction model, and then data scheduling is performed based on the predicted load. However, the prediction model is usually based on big data training, such as machine learning, and its model accuracy depends on the accuracy of the model itself and the amount of data trained, resulting in a certain degree of uncertainty in the prediction results, which leads to a high degree of dependence of data scheduling on the accuracy of the model itself and the amount of data trained. Summary of the invention

[0003] The embodiments of the present application provide a data scheduling method, device and electronic device to solve the problem in the prior art that, in the process of scheduling data in a database, the degree of dependence on the accuracy of the model itself and the amount of trained data is high.

[0004] Correspondingly, an embodiment of the present application also provides a data scheduling device, an electronic device and a storage medium to ensure the implementation and application of the above method.

[0005] In order to solve the above problems, an embodiment of the present application discloses a data scheduling method, which includes:

[0006] Obtaining a scheduling parameter of a storage node in a first scheduling period; wherein the scheduling parameter is determined according to predicted load data of the first scheduling period; the predicted load data is predicted according to historical data; the historical data includes data of historical scheduling periods before the first scheduling period;

[0007] The data of the storage node is scheduled according to the scheduling parameters.

[0008] Optionally, before obtaining the scheduling parameters of the storage node in the first scheduling period, the method further includes:

[0009] Acquire the historical data of a first preset number of the historical scheduling periods;

[0010] According to the historical data, the predicted load data of a second preset number of prediction scheduling periods are predicted; the prediction scheduling period includes the first scheduling period;

[0011] The scheduling parameters of the predicted scheduling period are determined according to the predicted load data.

[0012] Optionally, predicting and obtaining the predicted load data for a second preset number of prediction scheduling periods according to the historical data includes:

[0013] Obtain a hotspot distribution matrix of hotspot areas in the historical data;

[0014] Determining a load distribution matrix according to the hotspot distribution matrix;

[0015] The predicted load data of each storage node is determined according to the load distribution matrix.

[0016] Optionally, determining a load distribution matrix according to the hotspot distribution matrix includes:

[0017] Determine a replica matrix in the hotspot distribution matrix that satisfies the constraint condition;

[0018] Obtaining a load matrix of the replica matrix that satisfies the constraint condition;

[0019] A load distribution matrix for each storage node of the load matrix is ​​determined.

[0020] Optionally, determining the scheduling parameters of the predicted scheduling period according to the predicted load data includes:

[0021] Determine system evaluation parameters of candidate scheduling parameters according to the predicted load data; the system evaluation parameters include load balancing index parameters and migration cost parameters;

[0022] According to the system evaluation parameters, a scheduling parameter among the candidate scheduling parameters is determined.

[0023] Optionally, determining the scheduling parameters among the candidate scheduling parameters according to the system evaluation parameters includes:

[0024] According to the first data relationship, a minimum value of the sum of the load balancing index parameter and the migration cost parameter is determined, and the candidate scheduling parameter corresponding to the minimum value is the scheduling parameter, and the first data relationship includes:

[0025]

[0026] Wherein, J(U) represents the system evaluation parameter, U i represents the input sequence of the i-th prediction scheduling cycle, k represents the second preset number; Y irepresents the load balancing index parameter of the i-th prediction scheduling period, Q represents the first weight of the load balancing index parameter; and W represents the second weight of the migration cost parameter.

[0027] Optionally, the scheduling parameters include an original storage node and a target storage node corresponding to the data scheduling operation;

[0028] The step of scheduling the data of the storage node according to the scheduling parameter includes:

[0029] Migrate the data of the original storage node to the target storage node.

[0030] The embodiment of the present application also discloses a data scheduling device, the device comprising:

[0031] A parameter acquisition module, used to acquire the scheduling parameters of the storage node in the first scheduling period;

[0032] Wherein, the scheduling parameters are determined according to the predicted load data of the first scheduling period; the predicted load data are predicted according to historical data; the historical data include data of historical scheduling periods before the first scheduling period;

[0033] A data scheduling module is used to schedule the data of the storage node according to the scheduling parameters.

[0034] An embodiment of the present application also discloses an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the data scheduling method shown in the first aspect of the present application is implemented.

[0035] The embodiments of the present application further disclose a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, one or more methods described in the embodiments of the present application are implemented.

[0036] The beneficial effects of the technical solution provided by the embodiment of the present application are:

[0037] In an embodiment of the present application, a scheduling parameter of a storage node in a first scheduling period is obtained; data of the storage node is scheduled according to the scheduling parameter; the scheduling parameter is determined according to the predicted load data of the first scheduling period, thereby combining the control process with the prediction process, fully considering the impact of the scheduling operation on the database system, and improving the robustness of the system.

[0038] Additional aspects and advantages of the embodiments of the present application will be partially given in the description below, which will become apparent from the description below, or will be learned through the practice of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The above and / or additional aspects and advantages of the present application will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

[0040] Figure 1 One of the flow charts of the data scheduling method provided in the embodiment of the present application;

[0041] Figure 2 The second flowchart of the data scheduling method provided in the embodiment of the present application;

[0042] Figure 3 Flow chart 3 of the data scheduling method provided in the embodiment of the present application;

[0043] Figure 4 A schematic diagram of the structure of a data scheduling device provided in an embodiment of the present application;

[0044] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The embodiments of the present application are described below in conjunction with the drawings in the present application. It should be understood that the implementation methods described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0046] It will be understood by those skilled in the art that, unless specifically stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the terms "including" and "comprising" used in the embodiments of the present application refer to that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation as other features, information, data, steps, operations, elements, components and / or combinations thereof supported by the technical field. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element may be directly connected or coupled to the other element, or it may refer to that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term, for example, "A and / or B" may be implemented as "A", or as "B", or as "A and B".

[0047] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0048] See also Figure 1, an embodiment of the present application provides a data scheduling method, optionally, the method is applied to a distributed database, and the method may include the following steps:

[0049] Step 101, obtaining scheduling parameters of a storage node in a first scheduling period;

[0050] Wherein, the scheduling parameters are determined based on the predicted load data of the first scheduling cycle; the predicted load data are predicted based on historical data; and the historical data include data of historical scheduling cycles before the first scheduling cycle.

[0051] The first scheduling period is the period in which data scheduling is to be performed at the current moment, and data scheduling is, for example, migrating data from the first storage node (store1) to the second storage node (store2). Scheduling parameters are, for example, the amount of data scheduled in each scheduling period.

[0052] The scheduling parameters are determined based on the predicted load data of the first scheduling period. For example, the predicted load data of the first scheduling period is predicted at a time before the first scheduling period, and then the scheduling parameters of the first scheduling period are obtained based on the predicted load data. The minimum scheduling parameter that causes the system control error is selected to minimize the impact of the scheduling on the database system. For example, the scheduling parameter that causes the smallest load change or the smallest migration cost is selected to reduce the shock to the database system caused by data scheduling and the impact on the performance of the database system.

[0053] The historical data includes data of historical scheduling cycles before the first scheduling cycle, such as data of C historical scheduling cycles before the first scheduling cycle, where C can be a small positive integer, such as a value not greater than 10, and does not require large training data for adjustment, so that the prediction process is more integrated with the scheduling process and has higher interpretability.

[0054] Step 102: Schedule the data of the storage node according to the scheduling parameters.

[0055] According to the scheduling parameters, the data of the storage nodes are scheduled to control the error of the system by combining the scheduling parameters; the scheduling parameters are determined according to the predicted load data to combine the control process with the prediction process, fully consider the impact of the scheduling operation on the database system, and improve the robustness of the system.

[0056] In this way, during the data scheduling process, the dependence on the prediction model is reduced, thereby reducing the dependence on the accuracy of the model itself and the amount of trained data.

[0057] Optionally, in the embodiment of the present application, before obtaining the scheduling parameters of the storage node in the first scheduling period, the method further includes the first step to the third step:

[0058] The first step is to obtain the historical data of a first preset number of historical scheduling periods;

[0059] The second step is to predict and obtain the predicted load data of a second preset number of prediction scheduling periods according to the historical data; the prediction scheduling period includes the first scheduling period;

[0060] The third step is to determine the scheduling parameters of the predicted scheduling period according to the predicted load data.

[0061] Before scheduling data according to scheduling parameters, first predict the load data of the future scheduling cycle based on historical data, and generate scheduling parameters based on the predicted load data. Considering the randomness of distributed database hotspots, based on historical data, predict the data of the second preset number of cycles in the future, and calculate the error to minimize the overall error; determine the input of the predicted scheduling cycle based on the scheduling parameters, and use the input of the next scheduling cycle as a control variable to adjust the system, thereby improving the prediction accuracy and control response effect.

[0062] In addition, a data scheduling operation usually has a certain long tail effect, which will affect the system performance of multiple scheduling cycles, thus causing a shock effect on the control of the system and responding to the system performance. In the embodiment of the present application, by predicting the data of multiple cycles, the impact of the data scheduling operation on multiple scheduling cycles is considered to reduce unnecessary scheduling and improve the rapid response performance of the system.

[0063] Optionally, the second step further comprises:

[0064] Obtaining a hotspot distribution matrix of hotspot areas in the historical data; for example, a distributed storage node reports the top 100 regions (regions) of the node and the current store hotspot index to the metadata management node through the GRPC protocol, and obtaining the hotspot distribution matrix from the metadata management node; optionally, the metadata management node can also filter the metadata to reduce detection errors;

[0065] According to the hotspot distribution matrix, a load distribution matrix is ​​determined, and the load distribution matrix records the region load distribution on each store;

[0066] According to the load distribution matrix, the predicted load data of each storage node is determined, that is, the sum of the region loads on all stores.

[0067] Optionally, in an embodiment of the present application, determining a load distribution matrix according to the hotspot distribution matrix includes:

[0068] The leader in the hotspot distribution matrix is ​​determined through the raft protocol. Usually, each hotspot distribution matrix includes at least two replica matrices. Therefore, it is necessary to select the leader region leader (also called "preferred replica matrix") in the replica matrix. For example, a leader is selected through the raft protocol, and the leader distribution of each region is recorded through the leader matrix.

[0069] Obtaining a load matrix of the preferred replica matrix; the load matrix records the read traffic parameters of each region;

[0070] A load distribution matrix of each storage node of the load matrix is ​​determined, and the load distribution matrix records the region load distribution on each store.

[0071] Optionally, a specific example is used below to introduce the data scheduling method provided in an embodiment of the present application, which mainly includes two stages: a prediction stage and a control stage.

[0072] See also Figure 2 , the prediction stage includes steps 201 to 205.

[0073] Step 201: Obtain the distribution matrix A of the hotspot areas in the historical data MN , where A MN As shown in Matrix 1:

[0074]

[0075] In the matrix 1, M represents rows, N represents columns, and each element in the matrix represents a storage node; 1 represents that there is a hotspot on the storage node, and 0 represents that there is no hotspot on the storage node.

[0076] Step 202, determine the distribution matrix A MN The preferred replica matrix in the region leader.

[0077] Select the distribution matrix L of the region leader MN , L MN Record the leader distribution of each region, L MN As shown in Matrix 2:

[0078]

[0079] Among them, the replica matrix satisfies the following constraint 1:

[0080]

[0081] P represents the default number of replicas, for example, P can be 3.

[0082] The leader satisfies constraint 2: Among them, xnor represents the "exclusive OR" operation.

[0083] Step 203, obtain the read traffic parameters of each region in the region leader to obtain the load matrix region query .

[0084] Record the read traffic parameters of each region; since read traffic is only performed on the region leader, the region query is a diagonal matrix, as shown in Matrix 3:

[0085]

[0086] Where diag represents a diagonal matrix.

[0087] Step 204: according to the load matrix region query , determine the load matrix of each storage node store of the region leader Record the region load distribution on each store and get the load matrix As shown in the following matrix 4:

[0088]

[0089] Among them, dot represents dot product.

[0090] Step 205: Count the load matrix In the prediction load data store of each store query , store query As shown in the following matrix 5:

[0091]

[0092] Optionally, in the embodiment of the present application, determining the scheduling parameters of the predicted scheduling period according to the predicted load data includes:

[0093] Determine system evaluation parameters of candidate scheduling parameters according to the predicted load data; the system evaluation parameters include load balancing index parameters and migration cost parameters;

[0094] According to the system evaluation parameters, a scheduling parameter among the candidate scheduling parameters is determined.

[0095] Load balancing index parameters such as system error, migration cost parameters such as CPU utilization and network bandwidth occupied by data scheduling operations, etc.; according to the load balancing index parameters and migration cost parameters, the impact of scheduling operations on the database system is fully considered. For distributed storage nodes, multiple scheduling parameters (i.e., parameters corresponding to multiple scheduling operations) can be preset, and then the system evaluation parameters corresponding to each (or each group) of scheduling parameters are calculated separately. The system evaluation parameters include load balancing index parameters and migration cost parameters. Generally, the smaller the value of the system evaluation parameter, the more stable it is, so the alternative scheduling parameter with a smaller system evaluation parameter can be selected as the final scheduling parameter.

[0096] Further, determining the scheduling parameters in the candidate scheduling parameters according to the system evaluation parameters includes:

[0097] According to the first data relationship, a minimum value of the sum of the load balancing index parameter and the migration cost parameter is determined, and the candidate scheduling parameter corresponding to the minimum value is the scheduling parameter, and the first data relationship includes:

[0098]

[0099] Wherein, J(U) represents the system evaluation parameter, U i represents the input sequence of scheduling parameters of the i-th forecast scheduling period, k represents the second preset number; Y i represents the load balancing index parameter of the i-th prediction scheduling period, represents the migration cost parameter, Q represents the first weight of the load balancing index parameter; W represents the second weight of the migration cost parameter.

[0100] The following first introduces the derivation process of the first data relationship:

[0101] The first step is based on the load mean square error minimum formula:

[0102]

[0103] Among them, X(t) represents the distribution matrix L of the tth scheduling period MN ; X(t+1) represents the distribution matrix L of the t+1th scheduling period MN ; Y(k) represents the load balancing index parameter;

[0104] A and B represent the identity matrix E respectively;

[0105] U represents a scheduling parameter, which may be a migration matrix generated by a data scheduling operation;

[0106] C represents the preset operator dot(ones(1, N), region_query).

[0107] D is preset to 0;

[0108] M represents the scheduling delay.

[0109] The second step is to obtain formula 2 based on formula 1:

[0110]

[0111] In the third step, a new formula for predicting region leaders is obtained, as shown in the following formula 3:

[0112] X(t)=[x(t+1|t)] T , [x(t+2|t)] T , ..., [x(t+k|t)] T (Formula 3)

[0113] Wherein, k represents a second preset number, that is, the number of prediction scheduling cycles, for example, 3; and T represents a transposed matrix.

[0114] The fourth step is to obtain the scheduling parameter U(t) according to the preset scheduling parameter formula (Formula 4):

[0115] U(t)=[u(t|t)] T , [u(t+1|t)] T , ..., [u(t+k-1|t)] T (Formula 4)

[0116] Substituting U(t) into the formula of X(t), we get:

[0117]

[0118] Among them, the scheduling parameter U(t) satisfies the following constraints 3 and 4:

[0119] Constraint 3:

[0120] Constraint 4:

[0121] Among them, scheduler_limit indicates the number of concurrent schedules at the same time, which is used to prevent the concurrent number of schedules at the same time from being too high and affecting the front-end business of the distributed database; store_limit indicates the number of schedules on a single store, which is used to prevent too many schedules on a certain store from causing excessive pressure on a single database.

[0122] Step 5: Based on the above steps, the first data relationship is obtained:

[0123]

[0124] Wherein, J(U) represents the system evaluation parameter, U i represents the input sequence of the i-th prediction scheduling period, and k represents the second preset number;

[0125] Y i represents the load balancing index parameter of the i-th prediction scheduling period, Q represents the first weight of the load balancing index parameter; and W represents the second weight of the migration cost parameter.

[0126] in, Minimize the load mean square error, that is, the load of each node in the cluster is as balanced as possible; Represents the migration cost. For example, migrating a region from store1 to store2 requires CPU and network bandwidth.

[0127] Referring to the previous example, see Figure 3 The aforementioned control stage includes steps 301 to 302.

[0128] Step 301: Determine load balancing index parameters according to the predicted load data.

[0129] Step 302, determining the scheduling parameters among the candidate scheduling parameters according to the system evaluation parameters; optionally, a heuristic algorithm may be used to solve the first data relationship to obtain a control input sequence U=u(t),u(t+1),u(t+2) indicated by the scheduling parameters.

[0130] The first input is then used as a control variable for scheduling, and this process is repeated to predict and control the hotspot distribution of the distributed database.

[0131] Specifically, the data scheduling method provided by the embodiment of the present application is introduced below with a specific example, for example, the distribution matrix A of the hot spot area MN for;

[0132]

[0133] but

[0134] And the load matrix

[0135] Then we get the load distribution matrix

[0136] Statistics on the store

[0137] For each of the multiple sets of candidate scheduling parameters, the system evaluation parameters are calculated as follows:

[0138] Take the migration of scheduling parameters from store1 to store2 as an example. Assume that there is no scheduling in the first M = 5 scheduling cycles, the reported data is the same, the scheduling delay is m = 5, the prediction time domain is k = 3, and the Q and W weights are set to the unit matrix. Then:

[0139] 1. Migration cost parameter: C(U)=dot(abs(U), ones(N, 1)); where abs(U) represents absolute value operation.

[0140] 2. The first data relationship:

[0141] At this time, it is necessary to calculate the scheduling parameters for the next three predicted scheduling periods (t+1, t+2, t+3).

[0142] Assume that the last three plans are as follows: scheduling is performed in the t+1 scheduling cycle, and no scheduling is performed in the t+2 and t+3 scheduling cycles. Considering that the scheduling is completed with a delay of one cycle, the hotspot distribution and leader distribution of the last three time periods can be calculated:

[0143]

[0144] and same; Since no scheduling is generated, its value is the same as same.

[0145] Because store query Only with L MN Therefore, it is only necessary to calculate the load distribution matrix at the change time (the time after t+1 delay of one cycle, that is, t+2 prediction scheduling cycle) Similarly, count the total load corresponding to the store

[0146] and The same is no longer counted, and The same is no longer calculated, so the previous term of the first data relationship in the period from t to t+3 can be calculated:

[0147]

[0148] Since the scheduling delay k = 1, only t+1 has the migration cost calculated, while t+2 and t+3 do not.

[0149] Based on the second term of the first data relation, the migration cost for the entire period is:

[0150]

[0151] Therefore, the minimum value of the first data relationship (overall expected curve) is as follows J(U)=21+4=25, that is, in the scheduling period of t+2, the scheduling parameter is migrated from store1 to store2.

[0152] Optionally, in an embodiment of the present application, the scheduling parameters include an original storage node and a target storage node corresponding to the data scheduling operation;

[0153] The step of scheduling the data of the storage node according to the scheduling parameter includes:

[0154] Migrate the data of the original storage node to the target storage node. Optionally, the scheduling parameters can be carried in the scheduling instructions. The scheduling instructions mainly include the region to be scheduled (scheduling target) and the scheduling parameters (indicating which node to migrate from, i.e., the scheduling direction). Optionally, the scheduling instructions (which may have multiple specific scheduling steps) can be sent to the scheduling target through the heartbeat method of GRPC. The scheduling target will be migrated to the target storage node according to the scheduling instructions and reported to the metadata node of the database at the same time.

[0155] In an embodiment of the present application, a scheduling parameter of a storage node in a first scheduling period is obtained; data of the storage node is scheduled according to the scheduling parameter; the scheduling parameter is determined according to the predicted load data of the first scheduling period, thereby combining the control process with the prediction process, fully considering the impact of the scheduling operation on the database system, and improving the robustness of the system.

[0156] Based on the same principle as the method provided in the embodiment of the present application, the embodiment of the present application also provides a data scheduling device, such as Figure 4 As shown, the device comprises:

[0157] The parameter acquisition module 401 is used to acquire the scheduling parameters of the storage node in the first scheduling period;

[0158] Wherein, the scheduling parameters are determined based on the predicted load data of the first scheduling cycle; the predicted load data are predicted based on historical data; and the historical data include data of historical scheduling cycles before the first scheduling cycle.

[0159] The first scheduling period is the period in which data scheduling is to be performed at the current moment, and data scheduling is, for example, migrating data from the first storage node (store1) to the second storage node (store2). Scheduling parameters are, for example, the amount of data scheduled in each scheduling period.

[0160] The scheduling parameters are determined based on the predicted load data of the first scheduling period. For example, the predicted load data of the first scheduling period is predicted at a time before the first scheduling period, and then the scheduling parameters of the first scheduling period are obtained based on the predicted load data. The minimum scheduling parameter that causes the system control error is selected to minimize the impact of the scheduling on the database system. For example, the scheduling parameter that causes the smallest load change or the smallest migration cost is selected to reduce the shock to the database system caused by data scheduling and the impact on the performance of the database system.

[0161] The historical data includes data of historical scheduling cycles before the first scheduling cycle, such as data of C historical scheduling cycles before the first scheduling cycle, where C can be a small positive integer, such as a value not greater than 10, and does not require large training data for adjustment, so that the prediction process is more integrated with the scheduling process and has higher interpretability.

[0162] The data scheduling module 402 is used to schedule the data of the storage node according to the scheduling parameters.

[0163] According to the scheduling parameters, the data of the storage nodes are scheduled to control the error of the system by combining the scheduling parameters; the scheduling parameters are determined according to the predicted load data to combine the control process with the prediction process, fully consider the impact of the scheduling operation on the database system, and improve the robustness of the system.

[0164] In this way, during the data scheduling process, the dependence on the prediction model is reduced, thereby reducing the dependence on the accuracy of the model itself and the amount of trained data.

[0165] In an optional embodiment, the device comprises:

[0166] A data acquisition module, configured to acquire the historical data of a first preset number of the historical scheduling periods before the parameter acquisition module 401 acquires the scheduling parameters of the storage node in the first scheduling period;

[0167] A data prediction module, configured to predict, based on the historical data, the predicted load data of a second preset number of prediction scheduling periods; the prediction scheduling period includes the first scheduling period;

[0168] A parameter generation module is used to determine the scheduling parameters of the predicted scheduling period according to the predicted load data.

[0169] In an optional embodiment, the data prediction module includes:

[0170] A first acquisition submodule is used to acquire a hotspot distribution matrix of hotspot areas in the historical data;

[0171] A first determination submodule is used to determine a load distribution matrix according to the hotspot distribution matrix;

[0172] A statistical submodule is used to determine the predicted load data of each storage node according to the load distribution matrix.

[0173] In an optional embodiment, the first determining submodule is used to:

[0174] Determine the leader in the hotspot distribution matrix through the raft protocol;

[0175] Obtaining a load matrix of a leader in the hotspot distribution matrix;

[0176] A load distribution matrix for each storage node of the load matrix is ​​determined.

[0177] In an optional embodiment, the parameter generation module includes:

[0178] A second determination submodule is used to determine system evaluation parameters of candidate scheduling parameters according to the predicted load data; the system evaluation parameters include load balancing index parameters and migration cost parameters;

[0179] A generating submodule is used to determine the scheduling parameters among the candidate scheduling parameters according to the system evaluation parameters.

[0180] In an optional embodiment, the generating submodule is used to:

[0181] According to the first data relationship, a minimum value of the sum of the load balancing index parameter and the migration cost parameter is determined, and the candidate scheduling parameter corresponding to the minimum value is the scheduling parameter, and the first data relationship includes:

[0182]

[0183] Wherein, J(U) represents the system evaluation parameter, U i represents the input sequence of the i-th prediction scheduling cycle, k represents the second preset number; Y i represents the load balancing index parameter of the i-th prediction scheduling period, Q represents the first weight of the load balancing index parameter; and W represents the second weight of the migration cost parameter.

[0184] In an optional embodiment, the scheduling parameters include an original storage node and a target storage node corresponding to the data scheduling operation;

[0185] The data scheduling module 402 is used for:

[0186] Migrate the data of the original storage node to the target storage node.

[0187] The data scheduling device provided in the embodiment of the present application can achieve Figures 1 to 3 To avoid repetition, the various processes implemented in the method embodiment are not described here.

[0188] The data scheduling device provided by the present application comprises a parameter acquisition module 401 which acquires the scheduling parameters of the storage node in the first scheduling period; a data scheduling module 402 schedules the data of the storage node according to the scheduling parameters; the scheduling parameters are determined according to the predicted load data of the first scheduling period, thereby combining the control process with the prediction process, fully considering the impact of the scheduling operation on the database system, and improving the robustness of the system.

[0189] The data scheduling device of the embodiment of the present application can execute the data scheduling method provided by the embodiment of the present application, and the implementation principle is similar. The actions performed by each module and unit in the data scheduling device in each embodiment of the present application correspond to the steps in the data scheduling method in each embodiment of the present application. For the detailed functional description of each module of the data scheduling device, please refer to the description in the corresponding data scheduling method shown in the previous text, which will not be repeated here.

[0190] Based on the same principle as the method shown in the embodiments of the present application, the embodiments of the present application also provide an electronic device, which may include but is not limited to: a processor and a memory; the memory is used to store a computer program; the processor is used to execute the data scheduling method shown in any optional embodiment of the present application by calling the computer program.

[0191] In an optional embodiment, an electronic device is also provided, such as Figure 5 As shown, Figure 5 The electronic device 5000 shown includes: a processor 5001 and a memory 5003. The processor 5001 and the memory 5003 are connected, such as through a bus 5002. Optionally, the electronic device 5000 may also include a transceiver 5004, which may be used for data interaction between the electronic device and other electronic devices, such as data transmission and / or data reception. It should be noted that in actual applications, the transceiver 5004 is not limited to one, and the structure of the electronic device 5000 does not constitute a limitation on the embodiments of the present application.

[0192] Processor 5001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 5001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0193] The bus 5002 may include a path to transmit information between the above components. The bus 5002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 5002 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0194] The memory 5003 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, without limitation herein.

[0195] The memory 5003 is used to store the computer program for executing the embodiment of the present application, and the execution is controlled by the processor 5001. The processor 5001 is used to execute the computer program stored in the memory 5003 to implement the steps shown in the above method embodiment.

[0196] Among them, electronic devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0197] An embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the aforementioned method embodiment can be implemented.

[0198] The embodiment of the present application also provides a computer program product, including a computer program, which can implement the steps and corresponding contents of the aforementioned method embodiment when executed by a processor.

[0199] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification and claims of this application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the numbers used in this way can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in the drawings.

[0200] It should be understood that, although each operation step is indicated by arrows in the flowchart of the embodiment of the present application, the implementation order of these steps is not limited to the order indicated by the arrows. Unless clearly stated herein, in some implementation scenarios of the embodiment of the present application, the implementation steps in each flowchart can be performed in other orders according to demand. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on actual implementation scenarios. Some or all of these sub-steps or stages may be executed at the same time, and each sub-step or stage in these sub-steps or stages may also be executed at different times respectively. In different scenarios of execution time, the execution order of these sub-steps or stages may be flexibly configured according to demand, and the embodiment of the present application does not limit this.

[0201] The above is only an optional implementation method for some implementation scenarios of the present application. It should be pointed out that for ordinary technicians in this technical field, without departing from the technical concept of the solution of the present application, other similar implementation methods based on the technical ideas of the present application are also within the protection scope of the embodiments of the present application.

Claims

1. A data scheduling method, characterized in that: include: Obtain scheduling parameters of the storage node in the first scheduling period; Wherein, the scheduling parameters are determined according to the predicted load data of the first scheduling period; the predicted load data are predicted according to historical data; the historical data include data of historical scheduling periods before the first scheduling period; Scheduling the data of the storage node according to the scheduling parameters; Before obtaining the scheduling parameters of the storage node in the first scheduling period, the method further comprises: Acquire the historical data of a first preset number of the historical scheduling periods; According to the historical data, the predicted load data of a second preset number of prediction scheduling periods are predicted; the prediction scheduling period includes the first scheduling period; Determining the scheduling parameters of the prediction scheduling period according to the predicted load data; The step of predicting and obtaining the predicted load data for a second preset number of prediction scheduling periods according to the historical data includes: Obtain a hotspot distribution matrix of hotspot areas in the historical data; Determine the leader in the hotspot distribution matrix through the raft protocol; Obtaining a load matrix of a leader in the hotspot distribution matrix; Determining a load distribution matrix for each storage node of the load matrix; The predicted load data of each storage node is determined according to the load distribution matrix.

2. The data scheduling method according to claim 1, characterized in that: The determining the scheduling parameters of the prediction scheduling period according to the prediction load data includes: Determine system evaluation parameters of candidate scheduling parameters according to the predicted load data; the system evaluation parameters include load balancing index parameters and migration cost parameters; According to the system evaluation parameters, a scheduling parameter among the candidate scheduling parameters is determined.

3. The data scheduling method according to claim 2, characterized in that: The step of determining the scheduling parameters among the candidate scheduling parameters according to the system evaluation parameters includes: Determine, according to the first data relationship, a minimum value of the sum of the load balancing index parameter and the migration cost parameter, the candidate scheduling parameter corresponding to the minimum value being the scheduling parameter; Wherein, the first data relationship includes: Wherein, J(U) represents the system evaluation parameter, U i represents the input sequence of scheduling parameters of the i-th forecast scheduling period, k represents the second preset number; Y i represents the load balancing index parameter of the i-th prediction scheduling period, Q represents the first weight of the load balancing index parameter; and W represents the second weight of the migration cost parameter.

4. The data scheduling method according to any one of claims 1 to 3, characterized in that: The scheduling parameters include an original storage node and a target storage node corresponding to the data scheduling operation; The step of scheduling the data of the storage node according to the scheduling parameter includes: Migrate the data of the original storage node to the target storage node.

5. A data scheduling device, characterized in that: include: A parameter acquisition module, used to acquire the scheduling parameters of the storage node in the first scheduling period; Wherein, the scheduling parameters are determined according to the predicted load data of the first scheduling period; the predicted load data are predicted according to historical data; the historical data include data of historical scheduling periods before the first scheduling period; A data scheduling module, used for scheduling the data of the storage node according to the scheduling parameters; A data acquisition module, configured to acquire the historical data of a first preset number of the historical scheduling periods before the parameter acquisition module acquires the scheduling parameters of the storage node in the first scheduling period; A data prediction module, configured to predict, based on the historical data, the predicted load data of a second preset number of prediction scheduling periods; the prediction scheduling period includes the first scheduling period; A parameter generation module, used to determine the scheduling parameters of the forecast scheduling period according to the forecast load data; Wherein, the data prediction module includes: A first acquisition submodule is used to acquire a hotspot distribution matrix of hotspot areas in the historical data; The first determination submodule is used to determine the leader in the hotspot distribution matrix through the raft protocol; obtain the load matrix of the leader in the hotspot distribution matrix; and determine the load distribution matrix of each storage node of the load matrix; A statistical submodule is used to determine the predicted load data of each storage node according to the load distribution matrix.

6. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the method according to any one of claims 1 to 4 is implemented when the processor executes the program.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 4 is implemented.

Images