Batch scheduling method, device, equipment and computer storage medium

Abstract

The present invention relates to the technical field of financial technology (Fintech), and discloses a batch scheduling method, which includes: determining a target application among all applications to be run in parallel, performing master-slave partition processing on all machines associated with the target application to determine the partition corresponding to each of the machines; obtaining the decision factor of each of the machines, and calculating the machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines; obtaining the total amount of data to be run by the target application, calculating the amount of shard data of each of the partitions according to the total amount of data and the machine weight of each of the machines, and performing batch scheduling according to the amount of data of each of the shards. The present invention also discloses a batch multi-active device, equipment and a computer storage medium. The present invention improves the scheduling efficiency of batch multi-active.

Description

Technical Field

[0001] The present invention relates to the field of financial technology (Fintech) technology, and in particular to a batch scheduling method, device, equipment and computer storage medium. Background Art

[0002] With the development of computer technology, more and more technologies (big data, distributed, blockchain, artificial intelligence, etc.) are applied in the financial field. The traditional financial industry is gradually transforming to financial technology (Fintech). However, due to the security and real-time requirements of the financial industry, higher requirements are also put forward for batch multi-site active technology. The current batch application only supports single active. When batch applications are applied in multiple computer rooms, the performance is too different from the application performance in the same computer room due to the remote cross-computer room access to the database. When multiple batches are enabled at the same time, the multi-site batch application across computer rooms has a database access delay of 10ms more than the host in the main room. Therefore, when running batches together, the batch application of remote cross-computer rooms will run longer than the batch application of the same computer room, causing the entire batch to run slower. Therefore, how to improve the scheduling efficiency of batch multi-site active has become an urgent problem to be solved. Summary of the invention

[0003] The main purpose of the present invention is to propose a batch scheduling method, device, equipment and computer storage medium, aiming to solve the technical problem of how to improve the scheduling efficiency of batch multi-activity.

[0004] To achieve the above object, the present invention provides a batch scheduling method, which comprises the following steps:

[0005] Determine a target application among all the applications to be run in parallel, and perform primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines;

[0006] Obtaining a decision factor of each of the machines, and calculating a machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines;

[0007] The total data volume to be run by the target application is obtained, the shard data volume of each partition is calculated according to the total data volume and the machine weight of each machine, and batch scheduling is performed according to the shard data volume.

[0008] Optionally, the step of obtaining the decision factor of each of the machines and calculating the machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines includes:

[0009] Traversing each of the machines in turn, obtaining the partition type, number of threads, memory information, and reading efficiency of the traversed machines, and extracting the region weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information, and the reading efficiency weight corresponding to the reading efficiency from a preset configuration table, wherein the decision factors include the partition type, the number of threads, the memory information, and the reading efficiency;

[0010] A weight product among the partition weight, the thread weight, the memory weight, and the read efficiency weight is calculated, and the weight product is used as the machine weight of the traversed machine.

[0011] Optionally, the step of calculating the shard data volume of each partition according to the total data volume and the machine weight of each machine includes:

[0012] Determine the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine;

[0013] Traverse the weight proportions of each of the machines, calculate the first product of the total data volume and the traversed machine weight proportions, and use the first product as the total partition data of the partition corresponding to the traversed machine weight proportions, and determine the shard data volume of each partition based on the total data of each partition.

[0014] Optionally, the step of determining the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine includes:

[0015] Calculate a partition machine weight of each partition based on the machine weight of each machine and the partition corresponding to each machine;

[0016] Calculate the sum of the machine weights of each partition, use the sum as the total machine weight, traverse the machine weights of each partition, calculate the first ratio value of the traversed partition machine weight and the total machine weight, and use the first ratio value as the machine weight ratio of the partition corresponding to the traversed partition machine weight.

[0017] Optionally, the step of determining the amount of shard data of each partition based on the total data of each partition includes:

[0018] Traversing each of the partitions, determining the total number of machines of all machines in the traversed partitions and the number of threads of each of the machines, calculating a second product of the total number of machines and the number of threads of the machines, and using the second product as the total number of shards;

[0019] The total traversed partition data corresponding to the traversed partition is determined based on the total data of each partition, a second ratio value of the traversed partition total data and the total number of shards is calculated, and the second ratio value is used as the shard data amount of the traversed partition.

[0020] Optionally, the step of performing batch scheduling according to the data volume of each shard includes:

[0021] Calculating a third product of the total number of shards of the backup partition in each of the partitions and the preset number of executable shards;

[0022] Determine the total partition data of the backup partition in each of the partitions, and if the data volume of the total partition data of the backup partition is less than or equal to the third product, perform batch scheduling according to the machines in the backup partition; or,

[0023] If the total amount of partition data of the backup partition is greater than the third product, a step of performing batch scheduling according to the amount of data of each shard is executed.

[0024] Optionally, after the step of performing batch scheduling according to the data volume of each shard, the following steps are included:

[0025] If there is a partition of the running machine that is a primary partition, register a monitoring batch service for the running machine, and detect whether the partition of the running machine is converted from the primary partition to the backup partition based on a preset time interval;

[0026] If the partition of the running machine is converted from a primary partition to a backup partition, the monitoring batch service for the partition of the running machine is canceled, and the steps of obtaining the decision factors of each machine and calculating the machine weight of each machine according to the preset configuration table and the decision factors of each machine are executed.

[0027] In addition, to achieve the above-mentioned purpose, the present invention further provides a batch multi-active device, the batch multi-active device comprising:

[0028] A determination module, used to determine a target application among all applications to be run in parallel, and perform primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines;

[0029] An acquisition module, used to acquire the decision factor of each of the machines, and calculate the machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines;

[0030] The running module is used to obtain the total data volume of the target application to be run, calculate the shard data volume of each partition according to the total data volume and the machine weight of each machine, and perform batch scheduling according to the shard data volume.

[0031] In addition, to achieve the above-mentioned purpose, the present invention also provides a batch multi-active device, which includes: a memory, a processor, and a batch multi-active program stored in the memory and executable on the processor, and when the batch multi-active program is executed by the processor, the steps of the batch scheduling method as described above are implemented.

[0032] In addition, to achieve the above-mentioned purpose, the present invention also provides a computer storage medium, on which a batch multi-active program is stored, and when the batch multi-active program is executed by a processor, the steps of the batch scheduling method as described above are implemented.

[0033] The present invention determines the target application in all applications to be run in parallel, performs master-slave partition processing on all machines associated with the target application to determine the partition corresponding to each machine; obtains the decision factor of each machine, and calculates the machine weight of each machine according to the preset configuration table and the decision factor of each machine; obtains the total data volume of the target application to be run, calculates the shard data volume of each partition according to the total data volume and the machine weight of each machine, and performs batch scheduling according to the shard data volume. By performing master-slave partition processing on all machines associated with the target application in all applications to be run in parallel, the partition corresponding to each machine is determined, and then the machine weight of each machine is calculated according to the decision factor of each machine, and the shard data volume of each partition is calculated according to the total data volume of the target application to be run and the weight of each machine, so as to perform batch scheduling processing according to the data volume of each shard, thereby avoiding the inconsistency of the running time of different machines in the prior art, causing the slowdown of the running time of the entire batch, and dividing the total data according to the machine weight, thereby ensuring the consistency of the running time of each machine, and improving the scheduling efficiency of batch multi-activity. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a schematic diagram of a batch multi-active device structure of a hardware operating environment involved in an embodiment of the present invention;

[0035] Figure 2 This is a flow chart of a first embodiment of a batch scheduling method of the present invention;

[0036] Figure 3 This is a schematic diagram of a device module of a batch multi-activity device of the present invention;

[0037] Figure 4 It is a flow chart of the batch scheduling method of the present invention;

[0038] Figure 5 A schematic diagram of the process of heartbeat reporting and recording in the batch scheduling method of the present invention;

[0039] Figure 6It is a comparative flow chart of the slicing method of the prior art and the slicing method of the present invention;

[0040] Figure 7 A schematic diagram of the process of data distribution in the batch scheduling method of the present invention;

[0041] Figure 8 It is a flow chart of the process after the primary-standby conversion in the batch scheduling method of the present invention.

[0042] The realization of the purpose, functional features and advantages of the present invention will be further explained in conjunction with embodiments and with reference to the accompanying drawings. DETAILED DESCRIPTION

[0043] It should be understood that the specific embodiments described herein are only used to explain the present invention, and are not used to limit the present invention.

[0044] like Figure 1 As shown, Figure 1 It is a schematic diagram of the structure of batch multi-active devices in the hardware operating environment involved in the embodiment of the present invention.

[0045] The batch multi-active device in the embodiment of the present invention may be a PC or a server device, on which a Java virtual machine runs.

[0046] like Figure 1 As shown, the batch multi-active device may include: a processor 1001, such as a CPU, a network interface 1004, a user interface 1003, a memory 1005, and a communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. The user interface 1003 may include a display screen (Display), an input unit such as a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. The network interface 1004 may optionally include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a stable memory (non-volatile memory), such as a disk memory. The memory 1005 may optionally be a storage device independent of the aforementioned processor 1001.

[0047] Those skilled in the art will understand that Figure 1 The batch multi-active device structure shown in the figure does not constitute a limitation of the device, and may include more or fewer components than shown in the figure, or a combination of certain components, or a different arrangement of components.

[0048] like Figure 1 As shown, the memory 1005 as a computer storage medium may include an operating system, a network communication module, a user interface module, and a batch multi-activity program.

[0049] exist Figure 1 In the batch multi-active device shown, the network interface 1004 is mainly used to connect to the background server and communicate data with the background server; the user interface 1003 is mainly used to connect to the client (user end) and communicate data with the client; and the processor 1001 can be used to call the batch multi-active program stored in the memory 1005 and execute the operations in the following batch scheduling method.

[0050] Based on the above hardware structure, an embodiment of the batch scheduling method of the present invention is proposed.

[0051] Reference Figure 2 , Figure 2 This is a flow chart of a first embodiment of a batch scheduling method of the present invention, wherein the method comprises:

[0052] Step S10, determining a target application among all applications to be run in parallel, and performing primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines;

[0053] Currently, existing batch programs cannot truly achieve multi-active in different locations. They can only achieve the release of off-site backup applications, but they are not started. When the off-site applications are started at the same time, they will participate in the batch pulling, as well as the file download and export. When reading and writing databases across IDCs (data centers), it will increase the time consumption. Therefore, a batch scheduling method is provided to achieve multi-active in N computer rooms and N centers, to achieve the joint execution of batch programs, and to automatically identify the dynamic expansion clusters in different computer rooms, and to automatically detect the performance of machines in different computer rooms in accessing databases across computer rooms. In this embodiment, a shard selection algorithm based on the characteristics of batch machines is also provided, so that the applications and databases in N locations are not in the same computer room, and the performance of the machines is guaranteed to be stable when executing batch applications. In addition, a machine feature weight adjustment algorithm based on prediction is also provided in this embodiment to achieve multi-active switching of database computer rooms without manual intervention to automatically switch the machine resource allocation for running batches. That is, refer to Figure 4 First, determine all the applications to be run in parallel in N centers in N computer rooms, and perform heartbeat reporting of the machine IP and DCN (Data Center Node) in each computer room, and start the application execution performance analysis program to analyze and feedback the performance of the machine, and then calculate the running weight according to the performance of the machine, calculate the reasonable sharding data volume according to the number of machines and weights, and shard according to the number of weights of each machine in the primary and backup partitions, and automatically identify the switching resource allocation when disaster recovery switching occurs, and re-report the heartbeat to their respective DCNs.

[0054] Therefore, in this embodiment, all applications to be run in parallel in N centers in N computer rooms are first determined, and each computer room performs a heartbeat report on the machine IP and DCN where it is located, such as Figure 5As shown, the heartbeat report record is made in the database according to all batch processing applications A in the data center node and stored in the database. That is, all machines associated with the target application in all applications are determined, that is, all machines that need to be applied when the target application is running are determined, and these machines are regarded as all machines associated with the target application, and then the primary and backup partitions of these machines are processed according to the target application to determine the partitions corresponding to each machine, such as primary partition, backup partition, or primary partition, same-city backup partition, remote backup partition, etc.

[0055] Step S20, obtaining the decision factor of each of the machines, and calculating the machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines;

[0056] Moreover, in this embodiment, the application execution performance analysis program of the target application is started to obtain the performance characteristics of each machine, that is, to obtain the decision factors of each machine, and when calculating the machine weight according to the decision factors, the method for all machines is the same, that is, to first obtain the decision factors of the machine (which may include partition type, number of threads, memory information and reading efficiency), as shown in Table 1.

[0057] Decision Factors Decision conditions IDC type (partition type) Master partition, Slave partition The number of CPU core multithreading (number of threads) Number of CPU threads Memory information Memory Data Database read efficiency (read efficiency) Time spent on the last 100 transactions

[0058] Table 1

[0059] When obtaining the weight of the partition type, that is, the weight of the IDC type, a new configuration table will be added to configure the current DCN, determine whether the machine belongs to the Master partition or the Slave partition, query through the interface provided by the publishing system, and modify the query result according to the user's needs, that is, query the weight corresponding to the partition type of the machine in the configuration table, and in this embodiment, the benchmark can be set to Master partition weight 10 and Slave partition weight 5. Part of the content of the configuration table can be shown in Table 2 below.

[0060] Partition Type Weight Master 10 Slave in the same city 8 Remote Slave 5

[0061] Table 2

[0062] When obtaining the weight of the number of threads, that is, the weight of the number of core multi-threads of the CPU, the number of CPS cores of the benchmark can be set to 4 in the configuration table, the number of cores of the standby machine can be set to 2, and the weight corresponding to the number of threads of the machine can be queried in the configuration table. In addition, in this embodiment, the performance of the machine is related to the computer model selected in the computer room, and the machine with poor CPU performance can be modified according to actual needs.

[0063] Part of the content of the configuration table may be shown in Table 3 below.

[0064] Number of threads Weight 4 4 2 2

[0065] Table 3

[0066] When obtaining the weight of the memory information, the benchmark memory can be set to 2G memory in the configuration table, and the weight corresponding to the memory information of the machine can be queried in the configuration table. Part of the content of the configuration table can be shown in Table 4 below.

[0067] Memory information Weight 4G 4 2G 2 1G 1

[0068] Table 4

[0069] When obtaining the weight of the read efficiency, that is, the weight of the database read efficiency, it is also necessary to first make corresponding settings in the configuration table. In addition, since the DCN application of the cross-machine room IDC accesses other IDC databases, there will be network verification, which takes about 10ms, and the time consumption of the IDC in the same machine room is about 2ms, which is a large difference. Therefore, in order to avoid the instability of the query, the average time can be used to count the time consumption of 100 queries in the database. The query of the same IDC takes an average of 2ms; the query of cross-IDC takes an average of 10ms, and it is counted in the configuration table, and then the weight corresponding to the machine's read efficiency is queried after the configuration table. Among them, part of the content of the configuration table can be shown in Table 5 below.

[0070] Reading efficiency (average number of reads per 100 records) Weight 2ms 100 5ms 40 20ms 10

[0071] Table 5

[0072] After obtaining the weights corresponding to each decision factor of the machine, the machine weight of the machine can be calculated. That is, machine weight = partition type weight * thread number weight * memory information weight * read efficiency weight. Among them, the partition type weight is the weight corresponding to the partition type, the thread number weight is the weight corresponding to the thread number, the memory information weight is the weight corresponding to the memory information, and the read efficiency weight is the weight corresponding to the read efficiency.

[0073] Moreover, in this embodiment, the machine weights of all machines can be calculated in the above manner to obtain the machine weights of each machine, such as Master weight = Master type weight * 4 thread number weight * 4G memory weight * 2ms read efficiency weight = 10 * 4 * 4 * 100 = 16000.

[0074] Weight of Slave in the same city = weight of Slave type in the same city * weight of 4 threads * weight of 4G memory * weight of 10ms reading efficiency = 8*4*4*40=5120.

[0075] Remote Slave weight = Remote Slave type weight * 2 thread number weight * 2 GB memory weight * 20 ms read efficiency weight = 5 * 2 * 2 * 10 = 200.

[0076] Step S30, obtaining the total data volume of the target application to be run, calculating the shard data volume of each partition according to the total data volume and the machine weight of each machine, and performing batch scheduling according to the shard data volume.

[0077] After calculating the machine weight of each machine, you also need to calculate the total weight of all the master and backup machines, that is, the sum of the machine weights of all machines associated with the target application. That is, Sum (total weight of all the master and backup machines) = Sum (total weight of all the Master machines) + Sum (total weight of all the Slave machines in the same city) + Sum (total weight of all the Slave machines in different locations).

[0078] Then calculate the partition weight ratio of each partition based on the total weight of all the primary and backup machines. For example:

[0079] Master machine weight ratio = Sum (Master machine weight) / Sum (total weight of all master and backup machines).

[0080] Weight ratio of slave machines in the same city = Sum (weight of slave machines in the same city) / Sum (total weight of all primary and backup machines).

[0081] Weight ratio of remote slave machines = Sum (weight of remote slave machines) / Sum (total weight of all primary and backup machines).

[0082] Then, the total partition data of each partition is calculated based on the total amount of data to be run by the target application, such as:

[0083] Sum (total data of Master partition) = Sum (total data volume) * Master machine weight ratio;

[0084] Sum (total data of Slave partitions in the same city) = Sum (total data volume) * weight ratio of Slave machines in the same city;

[0085] Sum (total data of remote slave partitions) = Sum (total data volume) * weight ratio of remote slave machines.

[0086] In order to achieve the minimum data volume control for each sub-shard and avoid the phenomenon that the total data is evenly distributed to each machine after adding slave machines, and more shard scheduling causes unnecessary resource overhead, different solutions will be adopted for different scenarios. For example, if there are 100 data in total and the resources have 100 sub-shards available for calling, then each sub-shard will execute only one data. In the sharding algorithm, the data volume of each sub-shard is controlled and cannot be less than 20 / shard; for the same 100 data, resources are only allocated to 5 shards, with 20 data in each shard. There are three scenarios with less data volume, namely scenario 1, the total data volume is only enough for the Slave partition to complete the sharding; scenario 2, the total data volume is only enough for the Slave partition and part of the Master partition to execute; scenario 3 has no data. And the principle of data sharding is that the Slave partition is executed first to verify that the machine environment of the Slave partition is available. To avoid the situation where data is only executed in the Master partition, and suddenly the master-slave switch is made to the Slave partition, and the Slave partition is found to be unavailable, causing problems with the execution of a large amount of data in the Slave partition, manual intervention is required to check the environment configuration environment. Therefore, for scenarios 1 and 2, the total amount of data in the Slave partition must not be less than the total number of Slave shards * 20. If it is less than this data, all data is executed in the Slave partition. Optional solution: The total number of Slave shards can be specially configured. If the amount of data is small, the number of shards for each Slave machine can be configured to 1, and each machine only runs 1 machine to ensure that each machine participates in data processing. The remaining data is executed by the Master data. In scenario 3, there is no data, and only one sub-shard task needs to be run empty, and the Master machine will participate in the execution by default.

[0087] When the total partition data of each partition is calculated and the machines in each partition need to be executed, the total shards of each partition can be calculated first, that is, total shards = number of threads of each machine in the partition * total number of machines in the partition. In addition, the number of threads of each machine in the partition is proportional to the number of CPU cores. Users can also adjust the number of threads of a partition through the configuration table according to their own needs. And the amount of data in each shard = total partition data / total shards. That is:

[0088] The amount of data in each shard of the Master partition = Sum (total data of the Master partition) / total number of Master shards;

[0089] The amount of data in each shard of the Slave partition in the same city = Sum (total data of the Slave partition in the same city) / total number of Slave shards in the same city;

[0090] The amount of data in each shard of the remote slave partition = Sum (total data of the remote slave partition) / total number of remote slave shards.

[0091] The data information of each shard calculated will be stored in the database, and the source of the data processed by each shard will be directly obtained from the database to avoid repeated calculations each time. The calculation is done once, and the data is guaranteed to be consistent globally. In addition, each machine will perform batch scheduling according to the amount of data calculated for each shard, that is, single-threaded execution of data and multi-threaded execution of data. Among them, for single-threaded data, the data will only be executed on the machine that starts the batch. In addition, batch system interaction is carried out through files, so that remote machines can run multi-shard and multi-thread batches; files need to be imported into the database before multi-shard processing; after processing, the data for downstream is queried from the database and exported to the local disk, and then pushed to the downstream system. If the data volume of the file is large, it will also require a large delay to import it in the Slave partition. The same principle of pull batch control, for the machine of the DCN standby, does not participate in the file receiving task, import task, and export task.

[0092] And in this embodiment, the scheduling tasks of the master and the standby are processed separately, that is, the Master scheduling queue: is only responsible for distributing the Master partition data to the Master machine; the Slave scheduling queue in the same city: is only responsible for distributing the Slave partition data in the same city to the Slave machine in the same city; the Slave scheduling queue in a different location: is only responsible for distributing the Slave partition data in a different location to the Slave machine in a different location. The scheduling method of batch scheduling in the partition can be to distribute to the machine in sequence according to the order of the tasks, that is, the machine where the Step subtask i is located = (the total number of machines in the partition corresponding to the shard sequence number i%Step). And in this embodiment, when multi-shard scheduling is parallel thread task execution, the machine set for querying the heartbeat report table should be consistent, and the heartbeat query of the database needs to query the returned result set in ascending order of IP. After filtering the Master Slave partition machines during the memory processing process, the result set is processed in ascending order. The sharding process may also cause machine crashes, and it is necessary to cache the machine set in the main process of the shard, save it in the JVM thread stack of the main thread, and refresh it for each multi-shard Step. For example, Figure 7 As shown, the batch processing application A in the client distributes data to the servers of each batch processing application A according to the logical sharding model.

[0093] In order to help understand the method of performing multiple sharding according to weights in this embodiment, an example is given below. Figure 6As shown, in the prior art, the shards are read in full at one time, and the shards are divided in proportion to the total shards, that is, the system throughput can be divided into 4 equal parts, each occupying 1 / 4. Each two equal parts are processed by one thread. In this example, after the transformation, the shards are read in full at one time, and the data volume of the primary and backup partitions is divided according to the DCN weight ratio, such as the primary and backup ratio of 3:1. The DCN of the primary and backup partitions is further divided according to the proportion of the allocated data volume. Compared with the prior art, its batch processing efficiency is higher.

[0094] In this embodiment, by determining the target application in all applications to be run in parallel, all machines associated with the target application are processed for primary and secondary partitions to determine the partitions corresponding to each of the machines; the decision factors of each of the machines are obtained, and the machine weights of each of the machines are calculated according to the preset configuration table and the decision factors of each of the machines; the total amount of data to be run by the target application is obtained, and the amount of shard data of each partition is calculated according to the total amount of data and the machine weights of each of the machines, and batch scheduling is performed according to the amount of data of each shard. By performing primary and secondary partition processing on all machines associated with the target application in all applications to be run in parallel, the partitions corresponding to each machine are determined, and then the machine weights of each machine are calculated according to the decision factors of each machine, and the amount of shard data of each partition is calculated according to the total amount of data to be run by the target application and the weights of each machine, so as to perform batch scheduling according to the amount of data of each shard, thereby avoiding the phenomenon that the running time of different machines is inconsistent in the prior art, causing the running time of the entire batch to be slowed down, and the total data is divided according to the machine weight, thereby ensuring the consistency of the running time of each machine, and improving the scheduling efficiency of batch multi-activity.

[0095] Further, based on the first embodiment of the batch scheduling method of the present invention, a second embodiment of the batch scheduling method of the present invention is proposed. This embodiment is a refinement of step S20 of the first embodiment of the present invention, obtaining the decision factor of each of the machines, and calculating the machine weight of each of the machines according to the preset configuration table and the decision factor of each of the machines, including:

[0096] Step a, traversing each of the machines in turn, obtaining the partition type, number of threads, memory information and reading efficiency of the traversed machines, extracting the area weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information and the reading efficiency weight corresponding to the reading efficiency from a preset configuration table, wherein the decision factors include the partition type, the number of threads, the memory information and the reading efficiency;

[0097] In this embodiment, the machine weight of each machine can be calculated by first traversing each machine in turn and obtaining various decision factors of the traversed machines, such as partition type, number of threads, memory information and reading efficiency, and then extracting a configuration table set in advance from the database, and extracting the weight corresponding to the partition type of the traversed machine in the configuration table, that is, the area weight; the weight corresponding to the number of threads, that is, the thread weight; the weight corresponding to the memory information, that is, the memory weight; the weight corresponding to the reading efficiency, that is, the reading efficiency weight.

[0098] Step b: Calculate the weight product among the partition weight, the thread weight, the memory weight and the read efficiency weight, and use the weight product as the machine weight of the traversed machine.

[0099] After obtaining the weights corresponding to each decision factor of the traversed machine, the machine weight of the machine can be calculated. That is, machine weight = partition type weight * thread number weight * memory information weight * read efficiency weight. In other words, the weight product between the partition weight, thread weight, memory weight and read efficiency weight is used as the machine weight of the traversed machine. And the machine weights of all machines can be calculated in the above way to obtain the machine weight of each machine.

[0100] In this embodiment, by traversing each machine, the partition type, number of threads, memory information and reading efficiency of the traversed machine are obtained, and the weight product of the partition weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information and the reading efficiency weight corresponding to the reading efficiency is calculated according to the configuration table, and the weight product is used as the machine weight of the traversed machine, thereby ensuring the validity of the obtained machine weight.

[0101] Furthermore, the step of calculating the shard data volume of each partition according to the total data volume and the machine weight of each machine includes:

[0102] Step c, determining the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine;

[0103] In this embodiment, when calculating the total partition data of each partition, you can first determine the partition where each machine is located, such as the primary partition, the backup partition, etc., and then calculate the machine weight ratio of each partition based on the machine weight of each machine in the different partitions. The specific calculation method can be described in steps e and f below and will not be elaborated here.

[0104] Step d, traverse the weight proportions of each of the machines, calculate the first product of the total data volume and the traversed machine weight proportions, and use the first product as the total partition data of the partition corresponding to the traversed machine weight proportions, and determine the shard data volume of each partition based on the total data of each partition.

[0105] After calculating the machine weight ratio of all machines, you can traverse the weight ratio of each machine and calculate the product of the total amount of data to be run by the target application and the traversed machine weight ratio, that is, the first product, and use the first product as the partition total data of the partition corresponding to the traversed machine weight ratio. And the partition total data of all partitions can be calculated in this way. Then, the shard data volume of each partition is determined based on the number of machines in each partition and the total data of each partition.

[0106] In this embodiment, the machine weight ratio of each partition is determined according to each machine weight and the partition corresponding to each machine, and each machine weight ratio is traversed. The first product of the total data volume and the traversed machine weight ratio, that is, the total partition data, is calculated. The shard data volume of each partition is calculated according to the total data of each partition, thereby ensuring the validity of the acquired shard data volume.

[0107] Specifically, the step of determining the machine weight proportion of each partition according to the machine weight of each machine and the partition corresponding to each machine includes:

[0108] Step e, calculating the partition machine weight of each partition based on the machine weight of each machine and the partition corresponding to each machine;

[0109] In this embodiment, when calculating the machine weight ratio of each partition, it is necessary to first determine the partition corresponding to each machine, and calculate the sum of the machine weights of each machine in each partition, and use it as the partition machine weight.

[0110] Step f, calculate the sum of the machine weights of each partition, use the sum as the total machine weight, traverse the machine weights of each partition, calculate the first ratio value of the traversed partition machine weight and the total machine weight, and use the first ratio value as the machine weight ratio of the partition corresponding to the traversed partition machine weight.

[0111] In this embodiment, after calculating the weights of each partition, it is necessary to calculate the sum of the weights of the machines in each partition, and use this sum as the total weight of the machine, that is, the total weight of all the machines in the master and standby, Sum (total weight of all the machines in the master and standby) = Sum (sum of the weights of all the machines in the Master) + Sum (sum of the weights of all the machines in the same city Slave) + Sum (sum of the weights of all the machines in different locations Slave). Then traverse the weights of the machines in each partition, calculate the ratio of the traversed partition machine weights and the total machine weight, that is, the first ratio value, and use this first ratio value as the ratio of the machine weights of the partition corresponding to the traversed partition machine weights, such as: Master machine weight ratio = Sum (Master machine weight) / Sum (total weight of all the machines in the master and standby). Slave machine weight ratio in the same city = Sum (Slave machine weight in the same city) / Sum (total weight of all the machines in the master and standby). Slave machine weight ratio in different locations = Sum (Slave machine weight in different locations) / Sum (total weight of all the machines in the master and standby).

[0112] In this embodiment, the partition machine weight of each partition is calculated according to each machine weight, and then the total machine weight is calculated, the machine weights of each partition are traversed, and the first ratio value of the traversed partition machine weights and the total machine weight is calculated to determine the machine weight ratio, thereby ensuring the accuracy of the obtained machine weight ratio.

[0113] Furthermore, the step of determining the amount of shard data of each partition based on the total data of each partition includes:

[0114] Step g, traversing each of the partitions, determining the total number of machines of all machines in the traversed partitions and the number of threads of each of the machines, calculating the second product of the total number of machines and the number of threads of the machines, and taking the second product as the total number of shards;

[0115] In this embodiment, after obtaining the total data of each partition, each partition can be traversed to determine the total number of machines (i.e., the total number of machines) and the number of threads of each machine in the traversed partition, and the second product of the total number of machines and the number of threads of the machine is calculated, and the second product is used as the total number of shards. That is, the total number of shards = the number of threads of each machine in the partition (i.e., the number of threads of the machine) * the total number of machines.

[0116] Step h, based on the total data of each partition, determine the total data of the traversed partition corresponding to the traversed partition, calculate the second ratio value of the total data of the traversed partition and the total number of shards, and use the second ratio value as the shard data amount of the traversed partition.

[0117] In this embodiment, it is also necessary to determine the total partition data corresponding to the traversed partition in the total data of each partition, that is, the traversed partition total data, and then calculate the ratio of the traversed partition total data to the total number of shards, that is, the second ratio value, and use this second ratio value as the shard data volume of each shard in the traversed partition, such as: the data volume of each shard of the Master partition = Sum (total data of the Master partition) / total number of Master shards; the data volume of each shard of the Slave partition in the same city = Sum (total data of the Slave partition in the same city) / total number of Slave shards in the same city; the data volume of each shard of the Slave partition in a different location = Sum (total data of the Slave partition in a different location) / total number of Slave shards in a different location.

[0118] In this embodiment, by traversing each partition, the second product of the total number of machines in the traversed partition and the number of threads of the machine is calculated to determine the total number of shards, the total traversed partition data corresponding to the traversed partition is determined, and the second ratio value of the total traversed partition data and the total number of shards is calculated to determine the shard data volume of the traversed partition, thereby ensuring the accuracy of the acquired shard data volume.

[0119] Furthermore, the step of performing batch scheduling according to the data volume of each shard includes:

[0120] Step k, calculating the third product of the total number of shards of the backup partition in each of the partitions and the preset number of executable shards;

[0121] In this embodiment, it is necessary to determine the total number of shards in the backup partition from the calculated total number of shards, set a certain number of shard executables for each shard in the backup partition (that is, the number that each shard can execute), and calculate the third product of the total number of shards in the backup partition and the preset number of shard executables.

[0122] Step m, determining the total partition data of the backup partition in each of the partitions, and if the amount of the total partition data of the backup partition is less than or equal to the third product, performing batch scheduling according to the machines in the backup partition; or

[0123] After calculating the third product, it is also necessary to determine the total partition data of the backup partition in each partition, and then check whether the total partition data in the backup partition is less than or equal to the third product. If it is less than or equal to the third product, all data of the target application can be placed in the backup partition for execution, that is, the machines in the backup area are decomposed for batch scheduling.

[0124] Step n: if the total amount of partition data of the backup partition is greater than the third product, execute the step of batch scheduling according to the amount of data of each shard.

[0125] When it is determined that the total amount of partition data of the backup partition is greater than the third product, the step of batch scheduling according to the amount of data of each shard can be continued, that is, all partition machines participate in batch scheduling together.

[0126] In this embodiment, by calculating the third product of the total number of shards of the backup partition and the preset number of executable shards, and when the amount of total partition data of the backup partition is less than or equal to the third product, batch scheduling is directly performed according to the machines in the backup partition; when the amount of total partition data of the backup partition is greater than the third product, the step of batch scheduling according to the amount of data of each shard is continued, thereby ensuring the normal progress of batch scheduling.

[0127] Further, after the step of batch scheduling according to the data volume of each shard, the method further comprises:

[0128] Step p: if the partition of the running machine is a primary partition, register a monitoring batch service for the running machine, and detect whether the partition of the running machine is converted from a primary partition to a backup partition based on a preset time interval;

[0129] In this embodiment, the machine in the backup partition listens to the scheduling MQ queue of batch tasks, and the task message of pulling the batch is randomly sent to one of all the listening machines through the MQ queue; as long as the Slave machine does not listen to the queue of pulling the batch, the machine that can control the batch processing is on the Master partition. Therefore, when the target application is started, it will determine whether the partition where the current machine (i.e., the running machine) is located is the Master partition (i.e., the primary partition). If so, it will register the running machine to listen to the batch service. If not, it will not participate in the registration and listening to the batch service. And it will also detect whether the partition of the running machine is converted from the primary partition to the backup partition according to the preset time interval (any time interval set in advance by the user, such as 1 minute).

[0130] Step q: If the partition of the running machine is converted from a primary partition to a backup partition, the monitoring batch service for the partition of the running machine is canceled, and the step of obtaining the decision factors of each machine and calculating the machine weight of each machine according to the preset configuration table and the decision factors of each machine is executed.

[0131] When it is found through judgment that the partition of the running machine is the primary partition and the monitoring batch service has not been successfully registered, the monitoring operation of the batch service will be initiated and the registration result will be updated. If the partition is the primary partition and has been successfully registered, it will be skipped and not processed. If the partition is the backup partition and has been successfully registered, the batch service monitoring will be canceled. If the partition is the backup partition and there is no registered service, it will be skipped and not processed. In addition, in order to prevent the batch service from not updating the registration and canceling the registration in time during the primary and backup switching process, a monitoring and verification mechanism for the batch is added. If the machine for the batch is not the primary partition, the batch will be directly rejected, and it will be prompted that the batch must be allowed to be started in the primary partition. And after determining that the switching of the primary and backup partitions has occurred, the steps of obtaining the decision factors of each machine and calculating the machine weight of each machine according to the preset configuration table and the decision factors of each machine will continue to be executed. In order to readjust the amount of shard data in each partition.

[0132] In this embodiment, by determining that the partition where the machine is running is the primary partition, registering a monitoring batch service for the running machine, and when the partition of the running machine is converted from the primary partition to the backup partition, canceling the monitoring batch service, and continuing to execute the step of re-obtaining the machine weight of each machine, it is ensured that batch scheduling can be performed even in the case of primary-backup conversion.

[0133] It should be noted that the batch scheduling method in the present invention also supports re-running after batch break, that is, all data generated during batch scheduling will be stored in the database to ensure that the amount of data after re-running after batch break is consistent. Moreover, the shard scheduling data execution is stateful. Each step of update saves the processing status to the database, including the total data volume, total shard data, the data volume of each sub-shard, the total data volume of the Master partition, the total shard data of the Master partition, the total data volume of the Slave partition, and the total shard data of the Slave partition. The data information record processed by each shard, each data processed by the Step, has a granularity of Chunk, and all primary key keys are retained in the YAK_KEY_CONTEXT table. Which data is currently executed by the Step. When the batch is broken, the sharding algorithm will be maintained according to the number of master and backup shards recorded before, and the sharding algorithm will be used for sharding. The scheduled machine is stateless, and the machine information is re-acquired for sharding. Which data is executed before the batch is broken, then continue to execute the step from this data.

[0134] And in this embodiment, the definition of the primary and backup partitions is determined by whether the database and the machine are in the same computer room. If the primary and backup databases switch during the batch execution of data in a batch application, the primary and backup machines will also switch together. After the off-site disaster recovery switch, the off-site IDC becomes the same-city IDC, and the same-city IDC becomes the off-site IDC, and the positions are swapped accordingly. After the weights are updated by polling every minute, the weights are swapped, and when scheduling multiple shards, the corresponding tasks are also swapped accordingly. For example, Figure 8As shown, it includes the computer room DCNIO1, DCNIO2, database IO1 (standby), database IO2 (master), APPIP1 (standby partition), APPIP2 (standby partition), APPIP3 (standby partition), APPIP4 (standby partition), APPIP5 (standby partition), APPIP6 (standby partition), APPIP7 (standby partition) and APPIP8 (standby partition), scheduling platform, scheduling queue, master partition and configuration table. When running batch multi-active with master-standby switching, if the DCN of the database in the standby partition is abnormal, after switching to the database with IO2 as the master DCN, the operation and maintenance will modify MasterDCN to IO2, and register the batch service through the scheduling queue. Different IDCs applied in the database do not participate in the batch service registration, and then daily switching and batch pulling are performed, and the multi-active DCN and weight are obtained through the configuration table.

[0135] Moreover, due to the different performance of the primary and backup machines after the switch, the corresponding weight information needs to be re-counted. The shard scheduling queue re-identifies the primary and backup machines, re-divides the data volume of each shard of the primary and backup partitions, and then re-initiates requests to the primary and backup machines through independent primary and backup partition scheduling queues to evoke threads to execute shard subtasks.

[0136] The present invention also provides a batch scheduling device, referring to Figure 3 , the batch scheduling device comprises:

[0137] A determination module A10 is used to determine a target application among all applications to be run in parallel, and perform primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines;

[0138] An acquisition module A20, used to acquire a decision factor of each of the machines, and calculate a machine weight of each of the machines according to a preset configuration table and the decision factor of each of the machines;

[0139] The running module A30 is used to obtain the total data volume to be run by the target application, calculate the shard data volume of each partition according to the total data volume and the machine weight of each machine, and perform batch scheduling according to the shard data volume.

[0140] Optionally, the acquisition module A20 is used to:

[0141] Traversing each of the machines in turn, obtaining the partition type, number of threads, memory information, and reading efficiency of the traversed machines, and extracting the region weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information, and the reading efficiency weight corresponding to the reading efficiency from a preset configuration table, wherein the decision factors include the partition type, the number of threads, the memory information, and the reading efficiency;

[0142] A weight product among the partition weight, the thread weight, the memory weight, and the read efficiency weight is calculated, and the weight product is used as the machine weight of the traversed machine.

[0143] Optionally, the module A30 is run to:

[0144] Determine the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine;

[0145] Traverse the weight proportions of each of the machines, calculate the first product of the total data volume and the traversed machine weight proportions, and use the first product as the total partition data of the partition corresponding to the traversed machine weight proportions, and determine the shard data volume of each partition based on the total data of each partition.

[0146] Optionally, the module A30 is run to:

[0147] Calculate a partition machine weight of each partition based on the machine weight of each machine and the partition corresponding to each machine;

[0148] Calculate the sum of the machine weights of each partition, use the sum as the total machine weight, traverse the machine weights of each partition, calculate the first ratio value of the traversed partition machine weight and the total machine weight, and use the first ratio value as the machine weight ratio of the partition corresponding to the traversed partition machine weight.

[0149] Optionally, the module A30 is run to:

[0150] Traversing each of the partitions, determining the total number of machines of all machines in the traversed partitions and the number of threads of each of the machines, calculating a second product of the total number of machines and the number of threads of the machines, and using the second product as the total number of shards;

[0151] The total traversed partition data corresponding to the traversed partition is determined based on the total data of each partition, a second ratio value of the traversed partition total data and the total number of shards is calculated, and the second ratio value is used as the shard data amount of the traversed partition.

[0152] Optionally, the module A30 is run to:

[0153] Calculating a third product of the total number of shards of the backup partition in each of the partitions and the preset number of executable shards;

[0154] Determine the total partition data of the backup partition in each of the partitions, and if the data volume of the total partition data of the backup partition is less than or equal to the third product, perform batch scheduling according to the machines in the backup partition; or,

[0155] If the total amount of partition data of the backup partition is greater than the third product, a step of performing batch scheduling according to the amount of data of each shard is executed.

[0156] Optionally, the module A30 is run to:

[0157] If there is a partition of the running machine that is a primary partition, register a monitoring batch service for the running machine, and detect whether the partition of the running machine is converted from the primary partition to the backup partition based on a preset time interval;

[0158] If the partition of the running machine is converted from a primary partition to a backup partition, the monitoring batch service for the partition of the running machine is canceled, and the steps of obtaining the decision factors of each machine and calculating the machine weight of each machine according to the preset configuration table and the decision factors of each machine are executed.

[0159] The methods executed by the above-mentioned program units can refer to the various embodiments of the batch scheduling method of the present invention, which will not be described in detail here.

[0160] The invention also provides a computer storage medium.

[0161] The computer storage medium of the present invention stores a batch multi-active program, and when the batch multi-active program is executed by a processor, the steps of the batch scheduling method described above are implemented.

[0162] Among them, the method implemented when the batch multi-active program running on the processor is executed can refer to the various embodiments of the batch scheduling method of the present invention, and will not be repeated here.

[0163] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or system. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or system including the element.

[0164] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0165] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above-mentioned embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a number of instructions for a terminal device (which can be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0166] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A batch scheduling method, characterized in that: The batch scheduling method comprises the following steps: Determine a target application among all the applications to be run in parallel, and perform primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines; Traversing each of the machines in turn, obtaining the partition type, number of threads, memory information, and read efficiency of the traversed machines, and extracting the region weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information, and the read efficiency weight corresponding to the read efficiency from a preset configuration table; Calculate the weight product of the partition weight, the thread weight, the memory weight and the read efficiency weight, and use the weight product as the machine weight of the traversed machine; Obtaining the total amount of data to be run by the target application; Determine the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine; Traversing each of the machine weight proportions, calculating a first product of the total data volume and the traversed machine weight proportions, and using the first product as the partition total data of the partition corresponding to the traversed machine weight proportion, and determining the shard data volume of each partition based on the partition total data; Calculating a third product of the total number of shards of the backup partition in each of the partitions and the preset number of executable shards; Determine the total partition data of the backup partition in each of the partitions, and if the data volume of the total partition data of the backup partition is less than or equal to the third product, perform batch scheduling according to the machines in the backup partition; or, If the total amount of partition data of the backup partition is greater than the third product, batch scheduling is performed according to the amount of data of each shard.

2. The batch scheduling method according to claim 1, characterized in that: The step of determining the machine weight proportion of each partition according to the machine weight of each machine and the partition corresponding to each machine includes: Calculate a partition machine weight of each partition based on the machine weight of each machine and the partition corresponding to each machine; Calculate the sum of the machine weights of each partition, use the sum as the total machine weight, traverse the machine weights of each partition, calculate the first ratio value of the traversed partition machine weight and the total machine weight, and use the first ratio value as the machine weight ratio of the partition corresponding to the traversed partition machine weight.

3. The batch scheduling method according to claim 1, characterized in that: The step of determining the amount of shard data of each partition based on the total data of each partition comprises: Traversing each of the partitions, determining the total number of machines of all machines in the traversed partitions and the number of threads of each of the machines, calculating a second product of the total number of machines and the number of threads of the machines, and using the second product as the total number of shards; The total traversed partition data corresponding to the traversed partition is determined based on the total data of each partition, a second ratio value of the traversed partition total data and the total number of shards is calculated, and the second ratio value is used as the shard data amount of the traversed partition.

4. The batch scheduling method according to claim 1, characterized in that: After the step of performing batch scheduling according to the data volume of each shard, the method further includes: If there is a partition of the running machine that is a primary partition, register a monitoring batch service for the running machine, and detect whether the partition of the running machine is converted from the primary partition to the backup partition based on a preset time interval; If the partition of the running machine is converted from a primary partition to a backup partition, the monitoring batch service for the partition of the running machine is canceled, and the steps of obtaining the decision factors of each machine and calculating the machine weight of each machine according to the preset configuration table and the decision factors of each machine are executed.

5. A batch scheduling device, characterized in that: The batch scheduling device comprises: A determination module, used to determine a target application among all applications to be run in parallel, and perform primary and standby partition processing on all machines associated with the target application to determine a partition corresponding to each of the machines; an acquisition module, configured to traverse each of the machines in turn, acquire the partition type, number of threads, memory information and read efficiency of the traversed machines, extract the region weight corresponding to the partition type, the thread weight corresponding to the number of threads, the memory weight corresponding to the memory information and the read efficiency weight corresponding to the read efficiency from a preset configuration table; calculate the weight product among the partition weight, the thread weight, the memory weight and the read efficiency weight, and use the weight product as the machine weight of the traversed machine; A running module, used to obtain the total data volume to be run by the target application; determine the machine weight ratio of each partition according to the machine weight of each machine and the partition corresponding to each machine; traverse the machine weight ratios, calculate the first product of the total data volume and the traversed machine weight ratios, and use the first product as the partition total data of the partition corresponding to the traversed machine weight ratio, and determine the shard data volume of each partition based on the total data of each partition; calculate the third product of the total number of shards of the backup partition in each partition and the preset number of shard executables; Determine the total partition data of the backup partition in each of the partitions. If the amount of the total partition data of the backup partition is less than or equal to the third product, perform batch scheduling according to the machines in the backup partition; or, if the amount of the total partition data of the backup partition is greater than the third product, perform batch scheduling according to the data amount of each shard.

6. A batch scheduling device, characterized in that: The batch scheduling device comprises: a memory, a processor, and a batch scheduling program stored in the memory and executable on the processor. When the batch scheduling program is executed by the processor, the steps of the batch scheduling method according to any one of claims 1 to 4 are implemented.

7. A computer storage medium, characterized in that: The computer storage medium stores a batch scheduling program, and when the batch scheduling program is executed by the processor, the steps of the batch scheduling method according to any one of claims 1 to 4 are implemented.

Images