Data scheduling method, device, equipment, medium and system

Abstract

The invention provides a data scheduling method, device, equipment, medium and system, which can be used in the field of computers. The scheme is applied to a scheduling node in a data scheduling processing system, and the system comprises the scheduling node and a plurality of working nodes connected with the scheduling node. The method comprises the following steps: in response to a data scheduling task, performing scheduling task splitting to obtain a plurality of sub-scheduling tasks; extracting data corresponding to each sub-scheduling task from a database according to the plurality of sub-scheduling tasks; and distributing data corresponding to different sub-scheduling tasks to a plurality of working nodes for parallel processing. According to the data scheduling method, the data extraction task can be split into a plurality of sub-tasks for data parallel processing, so that the data processing time is effectively shortened.

Description

Technical Field

[0001] The present application relates to the field of computers, and in particular to a data scheduling method, device, equipment, medium and system. Background Art

[0002] Extracting data from databases is a critical task in daily enterprise operations. Through accurate data extraction, enterprises can gain insights into business operations, identify potential problems, and optimize resource allocation. With the rapid development of information technology, data sources, data format types, and data scale are increasing, making the need for efficient and accurate data extraction more and more significant.

[0003] At present, the data extraction method mainly adopts the serial processing mode, that is, data processing is performed on a single node, the required data is extracted from the database by executing the Structured Query Language (SQL), and then the exported data is processed sequentially according to specific logic to complete the data extraction.

[0004] However, the above data extraction method has a slow data processing speed when facing businesses with large data volumes and complex logic. Summary of the invention

[0005] The embodiments of the present application provide a data scheduling method, apparatus, device, medium and system to solve the problem of slow data processing speed of existing data extraction methods when facing businesses with large data volumes and complex logic.

[0006] In a first aspect, an embodiment of the present application provides a data scheduling method, which is applied to a scheduling node in a data scheduling processing system, wherein the system includes a scheduling node and a plurality of working nodes connected to the scheduling node, and the method includes:

[0007] In response to the data scheduling task, split the scheduling task to obtain multiple sub-scheduling tasks;

[0008] According to the multiple sub-scheduling tasks, extract the data corresponding to each sub-scheduling task from the database;

[0009] The data corresponding to different sub-scheduling tasks are distributed to multiple working nodes for parallel processing.

[0010] In a possible implementation, data corresponding to different sub-scheduling tasks are distributed to multiple working nodes for parallel processing, including:

[0011] According to the number of multiple working nodes or the current running status, the data corresponding to multiple sub-scheduling tasks are distributed to multiple working nodes for parallel processing.

[0012] In a possible implementation, according to the number of the plurality of working nodes, data corresponding to the plurality of sub-scheduling tasks are distributed to the plurality of working nodes for parallel processing, including:

[0013] According to the number of multiple working nodes and the number of multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the multiple working nodes for parallel processing.

[0014] In a possible implementation, according to the current operation status of the multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing, including:

[0015] According to the current operation status of multiple working nodes, the number of working nodes currently in idle state is determined, and according to the number of working nodes currently in idle state and the number of multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the idle working nodes for parallel processing;

[0016] or,

[0017] According to the current operation status of multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing according to the preset rules. The preset rules include: the difference in the amount of processed data of each working node after distribution is less than the preset value.

[0018] In a possible implementation, in response to the data scheduling task, the scheduling task is split to obtain a plurality of sub-scheduling tasks, including:

[0019] If the data scheduling task carries a scheduling time interval, the data scheduling task is split according to the preset scheduling time interval to obtain multiple sub-scheduling tasks;

[0020] or,

[0021] If the data scheduling task carries scheduling area information, the data scheduling task is split according to the preset scheduling area division to obtain multiple sub-scheduling tasks.

[0022] In a possible implementation, according to the multiple sub-scheduling tasks, extracting data corresponding to each sub-scheduling task from the database includes:

[0023] According to multiple sub-scheduling tasks, the data corresponding to each sub-scheduling task is extracted from the database in turn through SQL.

[0024] In a second aspect, an embodiment of the present application provides a data scheduling device, which is applied to a scheduling node in a data scheduling processing system, wherein the system includes a scheduling node and a plurality of working nodes connected to the scheduling node, and the device includes:

[0025] A scheduling task splitting module is used to split the scheduling task in response to the data scheduling task to obtain multiple sub-scheduling tasks;

[0026] A data extraction module is used to extract data corresponding to each sub-scheduling task from a database according to multiple sub-scheduling tasks;

[0027] The scheduling task allocation module is used to allocate the data corresponding to different sub-scheduling tasks to multiple working nodes for parallel processing.

[0028] In a third aspect, an embodiment of the present application provides a data scheduling device, including: a memory, a processor;

[0029] Memory stores computer-executable instructions;

[0030] The processor executes the computer-executable instructions stored in the memory, so that the processor executes the above first aspect and / or various possible implementations of the first aspect.

[0031] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, they are used to implement the first aspect above and / or various possible implementations of the first aspect.

[0032] In a fifth aspect, an embodiment of the present application provides a data scheduling processing system, including:

[0033] A scheduling node connected to a database, and multiple working nodes connected to the scheduling node;

[0034] Wherein, the scheduling node is used to execute the data scheduling method according to any one of claims 1 to 6;

[0035] The working nodes process the data assigned by the scheduling nodes.

[0036] The data scheduling method, device, equipment, medium and system proposed in the present application split the data processing task into multiple small tasks, and obtain the data corresponding to each small task from the corresponding database according to each split small task, and then distribute each small task and the corresponding data to different working nodes, so that each working node can process the corresponding data in parallel, thereby realizing that multiple working nodes can process data at the same time, thereby improving the speed of data processing, and parallel processing can better utilize data scheduling and processing resources, thereby achieving the effect of ensuring the efficient operation of the data scheduling and processing system. BRIEF DESCRIPTION OF THE DRAWINGS

[0037] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0038] Figure 1 A schematic diagram of a scenario of the data scheduling method provided by the present invention;

[0039] Figure 2 A flow chart of a first embodiment of a data scheduling method provided by the present invention;

[0040] Figure 3 A schematic diagram of a flow chart of a second embodiment of a data scheduling method provided by the present invention;

[0041] Figure 4 A schematic diagram of a flow chart of a third embodiment of a data scheduling method provided by the present invention;

[0042] Figure 5 A flowchart of a fourth embodiment of the data scheduling method provided by the present invention;

[0043] Figure 6 A flowchart of a fifth embodiment of the data scheduling method provided by the present invention;

[0044] Figure 7 A schematic diagram of the structure of the data scheduling device provided by the present invention;

[0045] Figure 8 A schematic diagram of the structure of the data scheduling device provided in this application.

[0046] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0048] With the rapid development of computer technology, data sources, data format types and data scale are getting larger and larger. When the data volume is huge and the processing logic is complex, the data extraction process often takes a lot of time, which affects the timeliness of business decisions. In addition, although some data extraction tasks can be split into multiple subtasks, in the case of serial execution, the optimization space is limited. Many data scheduling and processing systems fail to effectively schedule and utilize multi-node computing resources, resulting in resource waste.

[0049] In response to the above problems, when analyzing the data scheduling and processing method, the inventors thought of intelligently splitting the data extraction task into multiple independent sub-tasks according to the requirements or conditions of data extraction, and then evenly scheduling each sub-task to each working node according to the working conditions of each working node that processes the data, so that each working node can process the corresponding data in parallel, thereby greatly shortening the overall data extraction and processing time and achieving high efficiency of data extraction.

[0050] Figure 1 A schematic diagram of a scenario of the data scheduling method provided by the present invention, such as Figure 1 As shown, the specific application scenario of the present invention includes a data scheduling processing system 101 and a database 102, and the data scheduling processing system 101 includes a scheduling node 1011 and a working node 1012. Among them, the scheduling node 1011 is used to coordinate and schedule different data processing tasks, split the data scheduling task into multiple sub-scheduling tasks, and then extract the corresponding data from the database 102 according to each sub-scheduling task, and distribute it to multiple working nodes 1012 for data parallel processing to ensure that the dependencies between each sub-scheduling task are reasonably processed. The database 102 provides an efficient and secure data storage space for the data scheduling processing system 101, supports complex query and data analysis operations, and helps enterprises extract valuable information from massive data.

[0051] It should be noted that the data scheduling method, device, equipment, medium and system of the present application can be used in the computer field, and can also be used in any field other than the computer field. The application field of the data scheduling method, device, equipment, medium and system of the present application is not limited.

[0052] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described below in conjunction with the accompanying drawings.

[0053] Figure 2 Schematic diagram of the flow of the data scheduling method embodiment 1 provided by the present invention. Figure 2 As shown, the method is used for a scheduling node in a data scheduling processing system, the system includes a scheduling node, and a plurality of working nodes connected to the scheduling node, including:

[0054] S201. In response to a data scheduling task, split the scheduling task to obtain a plurality of sub-scheduling tasks.

[0055] Among them, a data scheduling task may refer to a request to extract data from a specific database, usually including data screening and aggregation. Scheduling task splitting may refer to breaking down a complex single task into multiple smaller independent tasks so that tasks can be processed in parallel on multiple nodes.

[0056] Exemplarily, after receiving a data scheduling task request, the scheduling node in the data scheduling processing system will first determine the complexity and data size of the data scheduling task, and then split the data scheduling task into multiple sub-scheduling tasks according to the complexity and data size of the task, wherein each sub-scheduling task is mainly for different data segments or processing tasks.

[0057] S202: According to the multiple sub-scheduling tasks, extract data corresponding to each sub-scheduling task from the database.

[0058] The database is a system used to store structured or unstructured data, such as a relational database or a distributed database. By using SQL, specific data can be retrieved from the database for subsequent data processing.

[0059] In a possible implementation, according to the multiple sub-scheduling tasks, extracting data corresponding to each sub-scheduling task from the database includes:

[0060] According to multiple sub-scheduling tasks, the data corresponding to each sub-scheduling task is extracted from the database in turn through SQL.

[0061] Specifically, the scheduling node in the data scheduling processing system generates corresponding SQL for each sub-scheduling task according to the split sub-scheduling tasks. After executing the corresponding SQL, the data set required for each sub-scheduling task can be extracted from the database in parallel.

[0062] S203: Allocate data corresponding to different sub-scheduling tasks to multiple working nodes for parallel processing.

[0063] Among them, parallel processing refers to executing multiple tasks on multiple working nodes at the same time, thereby improving the efficiency of data processing and reducing the total execution time of data processing.

[0064] Specifically, the scheduling node in the data scheduling processing system distributes the data extracted from the database to different working nodes according to the requirements of each sub-scheduling task. Each working node executes the corresponding data processing logic in parallel, such as data cleaning, format conversion, and data aggregation. Each working node processes the data set sent by the scheduling node separately, and transmits the processed data back to the scheduling node. The scheduling node merges and saves all the data for subsequent data analysis or problem troubleshooting.

[0065] The data scheduling method proposed in the embodiment of the present invention is applied to the scheduling node in the data scheduling processing system. By accepting requests for data extraction, each data scheduling task is analyzed and split into multiple sub-scheduling tasks, and a database query statement is generated for each sub-scheduling task to extract the required data from the database. The extracted data is then distributed to multiple working nodes for parallel processing, and the final data processing results are summarized, thereby improving the overall data processing efficiency. The method is particularly suitable for large-scale and complex data analysis scenarios.

[0066] Figure 3 Schematic diagram of the flow of the second embodiment of the data scheduling method provided by the present invention. Figure 3 As shown, in this embodiment Figure 2 Based on the embodiment, the method of allocating data corresponding to different sub-scheduling tasks to multiple working nodes for parallel processing is described in detail, and includes:

[0067] S301. Distribute data corresponding to multiple sub-scheduling tasks to multiple working nodes for parallel processing according to the number of multiple working nodes or the current operation status.

[0068] Among them, before allocating data, the scheduling node in the data scheduling processing system first needs to obtain the information of the available working nodes in the current system, such as the total number of working nodes available in the system, the load situation of each working node, including the number of tasks currently being processed, the number of completed tasks, and memory usage. According to the obtained working node information, the scheduling node needs to formulate a reasonable task allocation strategy. Possible strategies include load balancing strategy and priority strategy. Among them, the load balancing strategy tries to evenly distribute sub-scheduling tasks to each working node to ensure that no single node is overloaded, thereby improving the overall processing efficiency, while the priority strategy prioritizes tasks to idle or high-performance nodes based on the performance and current load situation of the working node. Finally, the scheduling node distributes the data corresponding to different sub-scheduling tasks to multiple working nodes according to the formulated task allocation strategy.

[0069] For example, according to the load balancing strategy, if there are 4 working nodes and 8 sub-scheduling tasks in the current data scheduling processing system, the scheduling node can allocate 2 sub-scheduling tasks to each working node. Once the data of the sub-scheduling tasks are allocated to each working node, all working nodes begin to process their respective sub-scheduling tasks in parallel. Each working node independently executes the corresponding data extraction and processing logic, thereby maximizing the use of multi-threaded parallel processing capabilities.

[0070] The data scheduling method provided in the embodiment of the present invention can effectively manage and allocate multiple sub-scheduling tasks through the scheduling nodes in the data scheduling processing system, make full use of the computing resources in the system, and realize the parallelization of data extraction and processing, thereby significantly improving the execution efficiency of the overall scheduling processing tasks in the case of large data volumes and complex data logic.

[0071] Figure 4 Schematic diagram of the flow of the data scheduling method embodiment 3 provided by the present invention. Figure 4 As shown, in this embodiment Figure 3 Based on the embodiment, the method of allocating data corresponding to multiple sub-scheduling tasks to multiple working nodes for parallel processing according to the number of multiple working nodes is described in detail. The method includes:

[0072] S401. According to the number of multiple working nodes and the number of multiple sub-scheduling tasks, evenly distribute the data corresponding to the multiple sub-scheduling tasks to the multiple working nodes for parallel processing.

[0073] In this embodiment, the scheduling nodes in the data scheduling processing system first need to be based on the number of available working nodes and the number of sub-scheduling tasks to be processed in the current system, where the above-mentioned number of working nodes and the number of sub-scheduling tasks can be obtained by querying the monitoring module and internal database in the data scheduling processing system.

[0074] Then, the scheduling node in the data scheduling processing system will calculate the number of tasks that should be assigned to each working node based on the above number of working nodes and the number of sub-scheduling tasks. For example, the number of tasks for each working node can be calculated by dividing the total number of sub-scheduling tasks by the number of working nodes.

[0075] Among them, if the number of sub-scheduling tasks cannot be divided by the number of working nodes, it is necessary to determine how to handle the remaining sub-scheduling tasks. For example, a "polling" method can be used to assign the excess tasks to the first few working nodes in turn to ensure that the tasks are distributed as evenly as possible. Specifically, the scheduling node can build a task allocation list based on the calculated number of tasks, where the task allocation list lists the index of the sub-scheduling tasks that each working node will process. For example, if there are 4 working nodes and 10 sub-scheduling tasks, the scheduling node can get the following allocation: Worker node 1 processes subtask 1 and subtask 5; Worker node 2 processes subtask 2 and subtask 6; Worker node 3 processes subtask 3 and subtask 7; Worker node 4 processes subtask 4 and subtask 8; Worker node 1 processes subtask 9; Worker node 2 processes subtask 10.

[0076] After generating the task assignment list, the scheduling node assigns the data corresponding to each sub-scheduled task to the corresponding worker node, where each worker node receives all the data required for the sub-scheduled task it is responsible for. After the data is assigned, each worker node starts to process its own sub-scheduled task in parallel, where each worker node independently executes the data extraction and processing logic, using its computing resources to efficiently process the assigned data.

[0077] After all working nodes complete data processing, the scheduling node will collect the processing results of each working node and perform necessary data integration, deduplication and formatting to form the final output data set. This process can ensure the integrity and consistency of the data processing results.

[0078] The data scheduling method provided in the embodiment of the present invention can realize efficient and average distribution of sub-scheduling tasks through the scheduling nodes in the data scheduling processing system, ensuring that the load of all working nodes is as balanced as possible, thereby optimizing the parallel processing efficiency and improving the speed and effect of overall data processing.

[0079] Figure 5 Schematic diagram of the flow of the fourth embodiment of the data scheduling method provided by the present invention. Figure 5 As shown, in this embodiment Figure 3 Based on the embodiment, according to the current operation status of the multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing, which includes:

[0080] S501. According to the current operation status of multiple working nodes, determine the number of working nodes that are currently in an idle state, and according to the number of working nodes that are currently in an idle state and the number of multiple sub-scheduling tasks, evenly distribute the data corresponding to the multiple sub-scheduling tasks to the idle working nodes for parallel processing.

[0081] In this step, the scheduling node in the data scheduling processing system first needs to obtain the operating status information of each working node through the system monitoring module, such as the number of tasks being processed by the working node, the usage of memory and storage, and whether the working node is currently processing tasks. Then, based on the acquired operating status information, the scheduling node will determine the number of working nodes in an idle state, where the idle state means that the working node currently has no tasks being executed, or the load is lower than a preset threshold, where the preset threshold is used to set standards and boundaries. For example, if the CPU (Central Processing Unit) usage or memory usage of a node exceeds a set threshold (e.g., 80%), the system will consider the node to be overloaded. In this case, the data scheduling processing system may avoid assigning new tasks to the node to prevent further increase in load and degradation of system performance.

[0082] After determining the number of idle working nodes, the scheduling node calculates according to the number of idle nodes and the number of sub-scheduling tasks to be processed, and determines the number of tasks that should be assigned to each idle node, for example, the number of tasks of each idle working node is equal to the number of sub-scheduling tasks divided by the number of idle working nodes. Among them, if the number of tasks cannot be divided evenly, the scheduling node can decide how to handle the remaining tasks, for example, according to the calculation results, a task allocation scheme is constructed, and the sub-scheduling tasks to be processed by each idle working node are listed in detail. Exemplarily, assuming that there are 6 idle working nodes and 14 sub-scheduling tasks, the task allocation may be as follows: working node 1 processes sub-task 1 and sub-task 12, working node 2 processes sub-task 2 and sub-task 11, working node 3 processes sub-task 3 and sub-task 10, working node 4 processes sub-task 4 and sub-task 9, working node 5 processes sub-task 5 and sub-task 8, working node 6 processes sub-task 6 and sub-task 7, working node 1 processes sub-task 12, working node 2 processes sub-task 13, and working node 3 processes sub-task 14.

[0083] Then, the scheduling node distributes the corresponding data to the idle working nodes according to the constructed task allocation plan. At this time, each working node starts to process the assigned sub-scheduling tasks in parallel.

[0084] S502: According to the current operation status of the multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes according to the preset rules for parallel processing.

[0085] Among them, the preset rules include that the difference in the amount of processed data of each working node after allocation is less than a preset value.

[0086] In this step, first, similar to step S501, the scheduling node obtains the current status of each working node, including its processing capacity and load. Then, before assigning task sub-scheduling tasks, the scheduling node needs to define preset rules, including the threshold of the difference in the amount of data processed by the working node, the minimum and maximum number of tasks, for example, the threshold of the difference in the amount of data processed by the working node can be set to 10%.

[0087] The scheduling node will then dynamically allocate tasks based on the acquired working node status and preset rules. For example, the current load of each working node and the maximum number of tasks that can be processed are calculated. Based on this information, the scheduling node attempts to allocate sub-scheduling tasks to nodes with lighter loads, ensuring that after allocation, the difference in the amount of data processed between the working nodes is less than the preset value. If there is a working node whose task volume is close to the maximum limit, the scheduling node can allocate new tasks to other working nodes with lighter loads.

[0088] After the task allocation is completed, all working nodes start parallel processing according to their assigned tasks to ensure maximum utilization of computing resources.

[0089] The data scheduling method provided by the embodiment of the present invention enables the scheduling node to effectively manage and allocate multiple sub-scheduling tasks, thereby ensuring load balancing and taking into account the current operation status of the working nodes. This data scheduling method can significantly improve the overall data processing efficiency and the system's responsiveness.

[0090] Figure 6 Schematic diagram of the flow of the fifth embodiment of the data scheduling method provided by the present invention. Figure 6 As shown, in this embodiment Figure 2 Based on the embodiment, in response to the data scheduling task, the scheduling task is split to obtain multiple sub-scheduling tasks, which are described in detail, including:

[0091] S601: If the data scheduling task carries a scheduling time interval, the data scheduling task is split according to a preset scheduling time interval to obtain a plurality of sub-scheduling tasks.

[0092] In this step, when the scheduling node receives the data scheduling task, it first checks whether the task contains the information of the scheduling time interval, which usually exists in the form of start time and end time, indicating the time range of the data to be extracted. Then, the scheduling node needs to obtain the preset scheduling time interval according to the system settings or configurations. This interval can be minutes, hours, days, etc., depending on the characteristics of the data and processing requirements. For example, the preset time interval may be one day.

[0093] Secondly, the scheduling node will split the time range into multiple sub-time intervals according to the obtained time interval and the preset time interval. For example, the total length of the time range is calculated. For example, if the start time is the data of January 1, 2023 and the end time is the data of December 31, 2023, the total length is 365 days. Then the total length is divided according to the preset time interval to obtain multiple sub-time intervals. For example, if the preset interval is one day, 365 sub-time intervals can be obtained.

[0094] According to each sub-time interval, the scheduling node generates a corresponding sub-scheduling task and the SQL corresponding to each sub-scheduling task for each time period, where each SQL carries the corresponding time interval information for subsequent data extraction and processing. Finally, the scheduling node returns the generated multiple sub-scheduling tasks and the SQL corresponding to each sub-scheduling task for subsequent parallel processing of data.

[0095] S602: If the data scheduling task carries scheduling area information, the data scheduling task is split according to the preset scheduling area division to obtain multiple sub-scheduling tasks.

[0096] In this step, after receiving the data scheduling task, the scheduling node will check whether the task contains regional information, where the regional information can be a specific geographical area, city, or specific district, etc. Then, the scheduling node needs to obtain the preset scheduling area division according to the system configuration, and then split the data scheduling task into multiple sub-scheduling tasks and generate corresponding SQL according to the obtained regional information and the preset area division.

[0097] Exemplarily, the scheduling node first determines the geographical scope involved in the data scheduling task. For example, the data scheduling task can be divided by province, state or city. Then the task is split into multiple subtasks, each of which corresponds to a specific geographical area. For example, if the task covers the whole country, it can be split into data extraction tasks for each province. For each divided area, the scheduling node generates a corresponding sub-scheduling task and corresponding SQL. Each SQL carries its specific geographical information so that the SQL can be distributed to different working nodes for parallel execution to quickly extract data.

[0098] In the data scheduling method proposed in the embodiment of the present invention, the scheduling node can effectively split the data scheduling task based on the scheduling time interval and regional information, thereby generating multiple sub-scheduling tasks and corresponding SQL, ensuring the flexibility and scalability of data processing, facilitating parallel processing in a large-scale data environment, and improving the overall data processing efficiency.

[0099] Figure 7 This is a schematic diagram of the structure of the data scheduling device provided by the present invention. Figure 7 As shown, the data scheduling device 700 includes:

[0100] The scheduling task splitting module 701 is used to split the scheduling task in response to the data scheduling task to obtain multiple sub-scheduling tasks;

[0101] A data extraction module 702 is used to extract data corresponding to each sub-scheduling task from a database according to multiple sub-scheduling tasks;

[0102] The scheduling task allocation module 703 is used to allocate data corresponding to different sub-scheduling tasks to multiple working nodes for parallel processing.

[0103] Optionally, the scheduling task allocation module 703 is specifically used for:

[0104] According to the number of multiple working nodes or the current running status, the data corresponding to multiple sub-scheduling tasks are distributed to multiple working nodes for parallel processing.

[0105] Optionally, the scheduling task allocation module 703 is further used to:

[0106] According to the number of multiple working nodes and the number of multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the multiple working nodes for parallel processing.

[0107] Optionally, the scheduling task allocation module 703 is further used to:

[0108] According to the current operation status of multiple working nodes, the number of working nodes currently in idle state is determined, and according to the number of working nodes currently in idle state and the number of multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the idle working nodes for parallel processing;

[0109] or,

[0110] According to the current operation status of multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing according to the preset rules. The preset rules include: the difference in the amount of processed data of each working node after distribution is less than the preset value.

[0111] Optionally, the scheduling task splitting module 701 is specifically used for:

[0112] If the data scheduling task carries a scheduling time interval, the data scheduling task is split according to the preset scheduling time interval to obtain multiple sub-scheduling tasks;

[0113] or,

[0114] If the data scheduling task carries scheduling area information, the data scheduling task is split according to the preset scheduling area division to obtain multiple sub-scheduling tasks.

[0115] Optionally, the data extraction module 702 is specifically used for:

[0116] According to multiple sub-scheduling tasks, the data corresponding to each sub-scheduling task is extracted from the database in turn through SQL.

[0117] The data scheduling device provided in the embodiment of the present invention can be used to execute the data scheduling method in any of the aforementioned method embodiments. Its implementation principle and technical effects are similar and will not be repeated here.

[0118] Figure 8 This is a schematic diagram of the structure of the data scheduling device provided by this application. Figure 8As shown, the data scheduling device 800 provided in this embodiment includes: at least one processor 801 and a memory 802. Optionally, the device 80 further includes a communication component 803. The processor 801, the memory 802 and the communication component 803 are connected via a bus 804.

[0119] In a specific implementation process, at least one processor 801 executes the computer-executable instructions stored in the memory 802, so that at least one processor 801 executes the above method.

[0120] The specific implementation process of the processor 801 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0121] In the above embodiments, it should be understood that the processor may be a CPU, or other general-purpose processors, digital signal processors (English: Digital Signal Processor, referred to as: DSP), application-specific integrated circuits (English: Application Specific Integrated Circuit, referred to as: ASIC), etc. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in the invention may be directly embodied as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0122] The memory may include a high-speed memory (Random Access Memory, RAM), and may also include a non-volatile memory (Non-volatile Memory, NVM), such as at least one disk memory.

[0123] The bus may be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0124] The present application also provides a data scheduling processing system, including a scheduling node connected to a database, and a plurality of working nodes connected to the scheduling node;

[0125] Wherein, the scheduling node is used to execute the data scheduling method of any one of the method embodiments;

[0126] The working nodes process the data assigned by the scheduling nodes.

[0127] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0128] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0129] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as: ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0130] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0131] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0132] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0133] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0134] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0135] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A data scheduling method, characterized in that: A scheduling node applied to a data scheduling processing system, wherein the system includes the scheduling node and a plurality of working nodes connected to the scheduling node, and the method includes: In response to the data scheduling task, split the scheduling task to obtain multiple sub-scheduling tasks; According to the multiple sub-scheduling tasks, extracting data corresponding to each sub-scheduling task from the database; The data corresponding to different sub-scheduling tasks are distributed to the multiple working nodes for parallel processing.

2. The method according to claim 1, characterized in that: The allocating data corresponding to different sub-scheduling tasks to the multiple working nodes for parallel processing includes: According to the number or current operation status of the multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing.

3. The method according to claim 2, characterized in that The allocating data corresponding to the plurality of sub-scheduling tasks to the plurality of working nodes for parallel processing according to the number of the plurality of working nodes includes: According to the number of the multiple working nodes and the number of the multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the multiple working nodes for parallel processing.

4. The method according to claim 2, characterized in that: The allocating data corresponding to the multiple sub-scheduling tasks to the multiple working nodes for parallel processing according to the current running conditions of the multiple working nodes includes: According to the current operation status of the multiple working nodes, the number of working nodes currently in an idle state is determined, and according to the number of working nodes currently in an idle state and the number of the multiple sub-scheduling tasks, the data corresponding to the multiple sub-scheduling tasks are evenly distributed to the idle working nodes for parallel processing; or, According to the current operation status of the multiple working nodes, the data corresponding to the multiple sub-scheduling tasks are distributed to the multiple working nodes for parallel processing according to preset rules, and the preset rules include: the difference in the amount of processed data of each working node after distribution is less than a preset value.

5. The method according to any one of claims 1 to 4, characterized in that: In response to the data scheduling task, the scheduling task is split to obtain multiple sub-scheduling tasks, including: If the data scheduling task carries a scheduling time interval, the data scheduling task is split according to a preset scheduling time interval to obtain the multiple sub-scheduling tasks; or, If the data scheduling task carries scheduling area information, the data scheduling task is split according to the preset scheduling area division to obtain the multiple sub-scheduling tasks.

6. The method according to any one of claims 1 to 4, characterized in that: The step of extracting data corresponding to each sub-scheduling task from a database according to the plurality of sub-scheduling tasks includes: According to the multiple scheduling sub-tasks, data corresponding to each scheduling sub-task is extracted from the database in sequence through the structured query language SQL.

7. A data scheduling device, characterized in that: A scheduling node applied to a data scheduling processing system, wherein the system includes the scheduling node and a plurality of working nodes connected to the scheduling node, and the device includes: A scheduling task splitting module is used to split the scheduling task in response to the data scheduling task to obtain multiple sub-scheduling tasks; A data extraction module, used to extract data corresponding to each sub-scheduling task from a database according to the multiple sub-scheduling tasks; The scheduling task allocation module is used to allocate the data corresponding to different sub-scheduling tasks to the multiple working nodes for parallel processing.

8. A data scheduling device, characterized in that: include: Memory, processor; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 6 when executed by a processor.

10. A data scheduling processing system, characterized in that: include: A scheduling node connected to a database, and a plurality of working nodes connected to the scheduling node; Wherein, the scheduling node is used to execute the data scheduling method according to any one of claims 1 to 6; The working node processes the data allocated to the scheduling node.

Images