Resource scheduling method and device, computer equipment and storage medium

Abstract

The invention relates to a resource scheduling method and device, computer equipment and a storage medium. The method comprises the following steps: acquiring a current data fragment in an offline data bin, calculating the number of data rows in the current data fragment, determining the number of nodes of a distributed computing framework according to the number of data rows in the current data fragment and the number of data rows processed by each node in the distributed computing framework, sending the number of nodes to the distributed computing framework, and sending the distributed computing framework to the offline data bin. Enabling the distributed computing framework to start a corresponding number of nodes, and synchronizing the data of the current data fragment to an online database through the started nodes. By adopting the method, the node number of the distributed computing framework can be dynamically adjusted, the dynamic scheduling of resources is realized, and the reasonable utilization of the resources is further realized.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a resource scheduling method, device, computer equipment and storage medium. Background Art

[0002] Offline features are usually generated by batch processing and extraction in a non-real-time environment, and the stored data in the offline environment is usually used for model training and analysis. If it needs to be provided to online real-time business queries, it needs to be synchronized to online storage to provide services.

[0003] In traditional technology, offline data is usually synchronized to online storage using a distributed computing system. During data processing, a distributed computing system can use multiple CPU computing nodes for parallel processing, but due to this feature, it often consumes a large amount of computing resources, including CPU and memory resources. When the amount of data is large, it is easy to cause unreasonable allocation of computing resources. Summary of the invention

[0004] Based on this, it is necessary to provide a resource scheduling method, apparatus, computer equipment and storage medium that can dynamically adjust computing resources according to the scale of data shards to address the above technical problems.

[0005] A resource scheduling method, comprising:

[0006] Get the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard;

[0007] Determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework;

[0008] The number of nodes is sent to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0009] In one embodiment, determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework includes:

[0010] Calculate the ratio of the number of rows in the current data shard to the number of data rows processed by each node;

[0011] When the ratio is less than the preset minimum value, the number of nodes is determined to be the preset minimum value;

[0012] When the ratio is greater than a preset maximum value, the number of nodes is determined to be the preset maximum value.

[0013] In one of the embodiments, the resource scheduling method further includes:

[0014] When the ratio is greater than or equal to a preset minimum value and less than or equal to a preset maximum value, the number of nodes is determined to be the ratio.

[0015] In one of the embodiments, the resource scheduling method further includes:

[0016] Monitor whether the current feature wide table generates new data shards;

[0017] When it is detected that the current feature wide table generates new data shards, a corresponding task is created for the new data shard and added to the pre-set priority queue;

[0018] Set weights for tasks;

[0019] The priority queue is sent to the distributed computing framework so that the distributed computing framework can obtain the corresponding data shards from the offline data warehouse to the online database according to the priority queue.

[0020] In one embodiment, setting weights for tasks includes:

[0021] Set the task weight according to the priority of the feature wide table to which the new data shard belongs.

[0022] In one embodiment, determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework includes:

[0023] The number of nodes in the distributed computing framework is calculated according to the following formula:

[0024]

[0025] Where E is the number of nodes in the distributed computing framework, E min is the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the weight of the feature wide table to which the current score shard belongs.

[0026] In one embodiment, the feature wide table to which the current data slice belongs is the current feature wide table, and the resource scheduling method further includes:

[0027] When obtaining the next data shard of the current feature wide table, the next data shard is used as the current data shard, and the process returns to the step of obtaining the current data shard in the offline data warehouse and calculating the number of data rows in the current data shard.

[0028] A resource scheduling device, comprising:

[0029] The acquisition module is used to obtain the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard;

[0030] A computing module, used to determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework;

[0031] The sending module is used to send the number of nodes to the distributed computing framework so that the distributed computing framework can start the corresponding number of nodes and synchronize the data of the current data shard to the online database through the started nodes.

[0032] A computer device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor implements the following steps when executing the computer program:

[0033] Get the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard;

[0034] Determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework;

[0035] The number of nodes is sent to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0036] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the following steps:

[0037] Get the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard;

[0038] Determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework;

[0039] The number of nodes is sent to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0040] The above-mentioned resource scheduling method, device, computer equipment and storage medium obtain the current data shard in the offline data warehouse, and calculate the number of data rows in the current data shard, determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, and send the number of nodes to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes, and synchronizes the data of the current data shard to the online database through the started nodes. The present application can calculate the number of nodes of the distributed computing framework according to the number of data rows of the data shard, so as to realize dynamic adjustment of the computing resources of the distributed computing framework, realize the reasonable allocation and utilization of resources, and improve resource utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 An application environment diagram of a resource scheduling method in an embodiment;

[0042] Figure 2 A schematic diagram of a resource scheduling method in one embodiment;

[0043] Figure 3 is a flow chart of a resource scheduling method in another embodiment;

[0044] Figure 4 is a structural block diagram of a resource scheduling device in one embodiment;

[0045] Figure 5 FIG. 4 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0047] Please refer to Figure 1 , Figure 1 A schematic diagram of an application environment of a resource scheduling method provided by an exemplary embodiment of the present application. Figure 1 As shown, the application environment includes an offline data warehouse 100, a scheduling server 101, a distributed computing framework 102 and an online database 103, wherein the distributed computing framework 102 includes multiple nodes (not shown in the figure), and the scheduling server 101 and the distributed computing framework 102 interact with data to implement the resource scheduling method of the present application.

[0048] The scheduling server 101 is used to obtain the current data shard in the offline data warehouse 100, and calculate the number of data rows in the current data shard, determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, and send the number of nodes to the distributed computing framework 102 so that the distributed computing framework 102 starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database 103 through the started nodes.

[0049] The distributed computing framework 102 is used to schedule the corresponding nodes according to the number of nodes sent by the scheduling server 101, and is used to synchronize the data in the offline data warehouse to the database 103, so as to dynamically schedule the corresponding computing resources for synchronizing data.

[0050] The scheduling server 101 may be implemented as an independent server or a server cluster consisting of multiple servers.

[0051] In one embodiment, Figure 2 As shown, a resource scheduling method is provided, which is applied to Figure 1 Taking the scheduling server in as an example, the resource scheduling method includes:

[0052] Step S21, obtain the current data shard in the offline data warehouse, and calculate the number of data rows in the current data shard.

[0053] Here, resources refer to computing resources, and specifically, resources refer to node resources in a distributed computing framework. Resource scheduling refers to calculating the number of nodes in a distributed computing framework according to the scale of data shards, thereby scheduling the corresponding number of nodes to achieve data synchronization.

[0054] The distributed computing framework can be Apache Spark, which is a powerful big data processing engine with the characteristics of fast and efficient, strong versatility and ease of use. It is often used in large-scale data processing such as offline features, that is, using Apache Spark's big data processing capabilities to synchronize offline storage to online storage systems.

[0055] Since the Apache Spark framework is a distributed computing system, it can use multiple CPU computing nodes for parallel processing in a distributed manner during data processing. However, due to such characteristics, when using Spark for data synchronization, a large amount of computing resources, including CPU and memory resources, are often consumed. Therefore, this application sets a reasonable number of nodes for the data synchronization task corresponding to each data shard by pre-calculating the number of nodes, and reasonably allocates computing resources on the premise of ensuring the timeliness of data synchronization to avoid affecting the stability of the online storage system.

[0056] Among them, the offline data warehouse is a data warehouse system that mainly uses batch processing to collect, store and analyze large amounts of historical data. It is usually used to update data regularly (such as daily, weekly, and monthly) to support various business analysis, report generation, and data mining tasks. The offline data warehouse is mainly aimed at scenarios where large-scale batch processing and analysis of historical data are performed. Compared with the real-time data warehouse, its core feature is that the data update and query response time is relatively long, which is suitable for business needs that do not require high real-time performance.

[0057] Specifically, the offline data warehouse includes multiple feature wide tables, and each feature wide table includes multiple data shards. Among them, the feature wide table is a data structure in which each row represents an observation or entity, and each column represents a characteristic or variable. This structure is usually very intuitive and easy to understand and operate the data. Data sharding is a data management technology that divides a large data set into multiple small blocks and stores them in different physical locations. When it is necessary to query or process data, the system can query or process only the corresponding data subset according to the rules of data sharding, thereby improving the efficiency and performance of data processing and query. Specifically, the data sharding of the present application can set the data of one day as a data shard, or it can set the data in other time periods as a data shard, which is not specifically limited here.

[0058] Step S22, determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework.

[0059] The distributed computing framework includes multiple nodes, each of which is used to perform data synchronization tasks. Specifically, the Apache Spark framework includes multiple worker nodes, each of which is used to synchronize data in the offline data warehouse to the online database.

[0060] The number of data rows processed by each node in the distributed computing framework is a preset value.

[0061] Step S23, sending the number of nodes to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0062] Among them, after the scheduling server calculates the number of nodes of the distributed computing framework, it sends the number of nodes to the distributed computing framework, so as to control the distributed computing framework to start the corresponding number of nodes to realize the scheduling of computing resources.

[0063] Exemplarily, when the scheduling server determines that the number of nodes is N, N is sent to the distributed computing framework, and the distributed computing framework correspondingly starts N nodes for synchronizing data.

[0064] It can be seen that the present application can dynamically adjust the number of nodes started by the distributed computing framework according to the number of data rows in the current data shard, so that the number of nodes is adapted to the scale of the data shard to be synchronized, thereby avoiding waste of resources, achieving rational use of resources, and improving resource utilization.

[0065] In one of the embodiments, the number of nodes of the distributed computing framework is determined based on the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, including: calculating the ratio of the number of rows in the current data shard to the number of data rows processed by each node, when the ratio is less than a preset minimum value, determining the number of nodes to be the preset minimum value, and when the ratio is greater than a preset maximum value, determining the number of nodes to be the preset maximum value.

[0066] Specifically, the present application may calculate the number of nodes of the distributed computing framework by the following calculation formula (1):

[0067]

[0068] Where E is the number of nodes in the distributed computing framework, E min is the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, and R is the number of data rows processed by each node.

[0069] The beneficial effect of this embodiment is: determining the number of nodes of the distributed computing framework according to the number of data rows of the current data shard in the offline data warehouse, so as to dynamically adjust the number of nodes of the distributed computing framework and realize the rational use of computing resources.

[0070] In one of the embodiments, the resource scheduling method may further include: when the ratio is greater than or equal to a preset minimum value and less than or equal to a preset maximum value, determining the number of nodes as the ratio.

[0071] In one of the embodiments, the resource scheduling method may also include: monitoring whether the current feature wide table produces new data shards. When it is monitored that the current feature wide table produces new data shards, creating corresponding tasks for the new data shards, adding them to a pre-set priority queue, setting weights for the tasks, and sending the priority queue to the distributed computing framework, so that the distributed computing framework obtains the corresponding data shards from the offline data warehouse to the online database according to the priority queue.

[0072] In this application, a priority queue is created in advance, and the priority queue is used to store the tasks to be processed and the priority of each task. Among them, each task corresponds to a data shard.

[0073] Specifically, the scheduling server can monitor the output of data shards. When it detects that new data shards are output, it creates corresponding tasks according to the new data shards and adds the tasks to the priority queue. Among them, each task in the priority queue can be sorted according to the time of joining and the priority. The priority includes the identification information, priority information and data shards corresponding to each task of each task. The scheduling server submits the identification information, priority information and data shards corresponding to each task to the distributed computing framework, so that the distributed computing framework obtains the corresponding data from the offline data warehouse according to the identification information, priority information and data shards corresponding to each task of the task for data synchronization.

[0074] The beneficial effect of this embodiment is that a weight value is set for each task so that tasks with high weights are processed preferentially according to the weight value.

[0075] In one of the embodiments, setting a weight for a task includes: setting the weight of the task according to a priority of a feature wide table to which the new data slice belongs.

[0076] In this application, the data in the offline data warehouse is divided into different feature wide tables according to different business scenarios, and corresponding priorities are set in advance for different feature wide tables according to the importance level of the business scenarios. Furthermore, the weights of the corresponding data shards are set according to the priorities of different feature wide tables.

[0077] In a possible design, the weights of the data slices corresponding to the same feature wide table may be set to be the same, or the weights of the data slices corresponding to the same feature wide table may be set to be different according to the priority of the feature wide table.

[0078] Exemplarily, assuming that the priority of feature wide table A is higher than that of feature wide table B, the weight of the data slice in feature wide table A is correspondingly set higher than the weight of the data slice in feature wide table B. The weights of different data slices in feature wide table A are the same or different.

[0079] In one embodiment, determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework includes:

[0080] The number of nodes in the distributed computing framework is calculated according to the following formula (2):

[0081]

[0082] Where E is the number of nodes in the distributed computing framework, E min is the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the priority of the feature wide table to which the current score shard belongs.

[0083] Specifically, the present application provides two methods for calculating the number of nodes, namely the above formula (1) and formula (2). The weight of each task is not included in formula (1), while the weight of each task is included in formula (2), so that the calculation result is more accurate.

[0084] Specifically, in formula (2), the scheduling server calculates the value of N / (R*W). When the ratio is less than the preset minimum value, the number of nodes is equal to the preset minimum value. When the value of N / (R*W) is greater than the preset maximum value, the number of nodes is equal to the preset maximum value. When the value of N / (R*W) is greater than or equal to the preset minimum value and less than or equal to the preset maximum value, the number of nodes is equal to N / (R*W).

[0085] In one of the embodiments, the feature wide table to which the current data shard belongs is the current feature wide table, and the method further includes: when obtaining the next data shard of the current feature wide table, taking the next data shard as the current data shard, returning to obtain the current data shard in the offline data warehouse, and calculating the number of data rows in the current data shard.

[0086] In this application, the offline data warehouse includes multiple feature wide tables, each feature wide table includes multiple data shards, and each data shard includes multiple data rows. When performing data synchronization, this application uses data shards as processing units, that is, the data of each data shard is used as the basis for resource scheduling.

[0087] Taking the current feature width table as an example, obtain the current data shard in the current feature width table, calculate according to the number of data rows in the current data shard, obtain the corresponding number of nodes, and submit it to the distributed computing framework to schedule the corresponding number of computing resources (ie, nodes).

[0088] When the next data shard is obtained, the corresponding number of nodes is calculated based on the number of data rows in the next data shard, and submitted to the distributed computing framework to schedule the corresponding number of computing resources. This allows the number of nodes in the distributed computing framework to be dynamically adjusted according to the scale of the data shards, rather than using the same number of nodes for each data shard, thus avoiding uneven resource utilization.

[0089] Please refer to Figure 3 , Figure 3 FIG. 1 is a flow chart of a resource scheduling method in an embodiment. Figure 3It includes offline data warehouse, scheduling server, Spark cluster and online storage. The offline data warehouse includes multiple feature wide tables. The scheduling server is used to calculate the number of nodes corresponding to the distributed computing framework according to the data sharding, so as to dynamically adjust the nodes in the distributed computing framework. The scheduling server also creates a priority queue to store the tasks and task weights corresponding to the data sharding. The Spark cluster is a distributed computing framework used to asynchronously consume the tasks in the priority queue and synchronize the data in the offline data warehouse to the online storage.

[0090] In one embodiment, Figure 4 As shown, a resource scheduling device is provided, including: an acquisition module 41, a calculation module 42 and a sending module 43, wherein:

[0091] The acquisition module 41 is used to acquire the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard;

[0092] A computing module 42, configured to determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework;

[0093] The sending module 43 is used to send the number of nodes to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0094] In one embodiment, the calculation module 42 can calculate the ratio of the number of rows in the current data shard to the number of data rows processed by each node. When the ratio is less than a preset minimum value, the number of nodes is determined to be the preset minimum value. When the ratio is greater than a preset maximum value, the number of nodes is determined to be the preset maximum value.

[0095] In one of the embodiments, the calculation module 42 may also determine the number of nodes as the ratio when the ratio is greater than or equal to a preset minimum value and less than or equal to a preset maximum value.

[0096] In one of the embodiments, the acquisition module 41 can also monitor whether the current feature wide table produces new data slices. When it is detected that the current feature wide table produces new data slices, a corresponding task is created for the new data slice, and it is added to a pre-set priority queue, and a weight is set for the task. The priority queue is sent to the distributed computing framework so that the distributed computing framework obtains the corresponding data slice from the offline data warehouse to the online database according to the priority queue.

[0097] In one embodiment, the acquisition module 41 may further: set the weight of the task according to the priority of the feature wide table to which the new data slice belongs.

[0098] In one embodiment, the calculation module 42 may calculate the number of nodes of the distributed computing framework according to the following formula:

[0099] Among them, E is the number of nodes in the distributed computing framework, Emin is the preset minimum value, Emax is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the weight of the feature wide table to which the current score shard belongs.

[0100] In one embodiment, the acquisition module 41 may also: when acquiring the next data shard of the current feature wide table, use the next data shard as the current data shard, return to the step of acquiring the current data shard in the offline data warehouse, and calculate the number of data rows in the current data shard.

[0101] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 5 As shown. The computer device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a resource scheduling method is implemented.

[0102] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented: obtaining a current data shard in an offline data warehouse, and calculating the number of data rows in the current data shard, determining the number of nodes in a distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework,

[0103] The number of nodes is sent to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

[0104] In one embodiment, when the processor executes a computer program to implement the above-mentioned step of determining the number of nodes of the distributed computing framework based on the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, the following steps are specifically implemented: calculating the ratio of the number of rows in the current data shard to the number of data rows processed by each node, when the ratio is less than a preset minimum value, determining the number of nodes to be the preset minimum value, and when the ratio is greater than a preset maximum value, determining the number of nodes to be the preset maximum value.

[0105] In one embodiment, when the processor executes the computer program, the following steps are specifically implemented: when the ratio is greater than or equal to a preset minimum value and less than or equal to a preset maximum value, the number of nodes is determined to be the ratio.

[0106] In one embodiment, when the processor executes the computer program, it also specifically implements the following steps: monitoring whether the current feature wide table produces new data shards, and when it is monitored that the current feature wide table produces new data shards, creating corresponding tasks for the new data shards, adding them to a pre-set priority queue, setting weights for the tasks, and sending the priority queue to the distributed computing framework, so that the distributed computing framework obtains the corresponding data shards from the offline data warehouse to the online database according to the priority queue.

[0107] In one embodiment, when the processor executes the computer program to implement the above-mentioned step of setting weights for tasks, the following steps are specifically implemented: setting the weight of the task according to the priority of the feature wide table to which the new data slice belongs.

[0108] In one embodiment, when the processor executes the computer program to implement the above-mentioned step of determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, the following steps are specifically implemented: the number of nodes of the distributed computing framework is calculated according to the following formula:

[0109]

[0110] Where E is the number of nodes in the distributed computing framework, E min is the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the weight of the feature wide table to which the current score shard belongs.

[0111] In one embodiment, the feature wide table to which the current data shard belongs is the current feature wide table. When the processor executes the computer program, the following steps are specifically implemented: when obtaining the next data shard of the current feature wide table, the next data shard is used as the current data shard, and the step of returning to obtain the current data shard in the offline data warehouse and calculating the number of data rows in the current data shard is performed.

[0112] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented: obtaining a current data shard in an offline data warehouse, and calculating the number of data rows in the current data shard, determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, sending the number of nodes to the distributed computing framework so that the distributed computing framework starts a corresponding number of nodes, and synchronizing the data of the current data shard to the online database through the started nodes.

[0113] In one embodiment, when a computer program is executed by a processor to implement the above-mentioned step of determining the number of nodes of a distributed computing framework based on the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, the following steps are specifically implemented: calculating the ratio of the number of rows in the current data shard to the number of data rows processed by each node, and when the ratio is less than a preset minimum value, determining the number of nodes to be the preset minimum value; when the ratio is greater than a preset maximum value, determining the number of nodes to be the preset maximum value.

[0114] In one embodiment, when the computer program is executed by a processor, the following steps are specifically implemented: when the ratio is greater than or equal to a preset minimum value and less than or equal to a preset maximum value, the number of nodes is determined to be the ratio.

[0115] In one embodiment, when the computer program is executed by the processor, the following steps are specifically implemented: monitoring whether the current feature wide table produces new data shards; when it is detected that the current feature wide table produces new data shards, creating corresponding tasks for the new data shards, adding them to a pre-set priority queue, setting weights for the tasks, and sending the priority queue to the distributed computing framework so that the distributed computing framework obtains the corresponding data shards from the offline data warehouse to the online database according to the priority queue.

[0116] In one embodiment, when the computer program is executed by the processor to implement the above-mentioned step of setting weights for tasks, the following steps are specifically implemented: setting the weight of the task according to the priority of the feature wide table to which the new data slice belongs.

[0117] In one embodiment, when the computer program is executed by the processor to implement the above-mentioned step of determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework, the following steps are specifically implemented: the number of nodes of the distributed computing framework is calculated according to the following formula:

[0118]

[0119] Where E is the number of nodes in the distributed computing framework, E minis the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the weight of the feature wide table to which the current score shard belongs.

[0120] In one embodiment, the feature wide table to which the current data shard belongs is the current feature wide table. When the computer program is executed by the processor, the following steps are specifically implemented: when obtaining the next data shard of the current feature wide table, the next data shard is taken as the current data shard, and the step of returning to obtain the current data shard in the offline data warehouse and calculating the number of data rows in the current data shard is performed.

[0121] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be implemented by instructing related hardware through a computer program, and the above-mentioned computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application may include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in many forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0122] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0123] The above embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A resource scheduling method, characterized in that: The method comprises: Obtain the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard; Determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework; The number of nodes is sent to the distributed computing framework so that the distributed computing framework starts a corresponding number of nodes, and synchronizes the data of the current data shard to the online database through the started nodes.

2. The method according to claim 1, characterized in that The determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework comprises: Calculate the ratio of the number of rows in the current data shard to the number of data rows processed by each node; When the ratio is less than a preset minimum value, determining the number of nodes to be the preset minimum value; When the ratio is greater than a preset maximum value, the number of nodes is determined to be the preset maximum value.

3. The method according to claim 2, characterized in that The method further comprises: When the ratio is greater than or equal to the preset minimum value and less than or equal to the preset maximum value, the number of nodes is determined to be the ratio.

4. The method according to claim 1, characterized in that: The method further comprises: Monitor whether the current feature wide table generates new data shards; When it is detected that the current feature wide table generates a new data slice, a corresponding task is created for the new data slice and added to a preset priority queue; Setting weights for the tasks; The priority queue is sent to the distributed computing framework so that the distributed computing framework obtains corresponding data shards from the offline data warehouse to the online database according to the priority queue.

5. The method according to claim 4, characterized in that The step of setting a weight for the task includes: The weight of the task is set according to the priority of the feature wide table to which the new data slice belongs.

6. The method according to claim 5, characterized in that The determining the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework comprises: The number of nodes in the distributed computing framework is calculated according to the following formula: Where E is the number of nodes in the distributed computing framework, E min is the preset minimum value, E max is the preset maximum value, N is the number of data rows in the current data shard, R is the number of data rows processed by each node, and W is the weight of the feature wide table to which the current score shard belongs.

7. The method according to claim 1, characterized in that The feature wide table to which the current data slice belongs is the current feature wide table, and the method further includes: When the next data shard of the current feature wide table is obtained, the next data shard is used as the current data shard, and the process returns to the step of obtaining the current data shard in the offline data warehouse and calculating the number of data rows in the current data shard.

8. A resource scheduling device, characterized in that: The device comprises: An acquisition module is used to acquire the current data shard in the offline data warehouse and calculate the number of data rows in the current data shard; A computing module, used to determine the number of nodes of the distributed computing framework according to the number of data rows in the current data shard and the number of data rows processed by each node in the distributed computing framework; The sending module is used to send the number of nodes to the distributed computing framework so that the distributed computing framework starts the corresponding number of nodes and synchronizes the data of the current data shard to the online database through the started nodes.

9. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.

Images