Distributed computing resource coordination task scheduling computing platform and method

Abstract

The invention discloses a distributed computing resource coordination task scheduling computing platform and method. Through intelligent resource scheduling and task allocation, efficient utilization of computing resources is realized, and smooth completion of data processing tasks is ensured. The platform comprises a scheduling coordination module, a job module, a data processing plug-in, a process engine and external middleware. The scheduling coordination module is fixed, is irrelevant to business, realizes decentration and high availability in design, and starts, monitors and closes a controlled process; the operation module is personalized, the data processing plug-in is externally provided with a standardized interface, and internal logic is related to a data processing algorithm. According to the invention, distributed computing resources can be effectively coordinated and managed, and efficient scheduling of tasks is realized; meanwhile, in consideration of the difficulty degree of platform deployment, the platform has the characteristics of simple structure, light operation weight, no complex component, no installation and the like.

Description

Technical Field

[0001] The present invention belongs to the data processing technology in the field of electromagnetic signal data processing, and in particular provides a distributed computing resource coordination task scheduling computing platform and method for large-scale, time-consuming data processing tasks to ensure the safety and efficient use of electromagnetic signal data processing. Background Art

[0002] As the electromagnetic environment becomes increasingly complex, the electromagnetic signal data generated every second is becoming increasingly complex and large in scale. Traditional data processing methods can no longer meet the needs of efficiency and real-time. Distributed computing, as an effective solution, can significantly improve data processing capabilities by distributing computing tasks on multiple computing nodes for parallel processing. However, how to effectively coordinate and manage distributed computing resources and achieve efficient task scheduling has become a problem that needs to be solved urgently. At the same time, considering the difficulty of platform deployment, the platform is required to have a simple structure, lightweight operation, no complex components, and no installation. Summary of the invention

[0003] The present invention proposes a distributed computing resource coordination task scheduling computing platform and method, which realizes efficient utilization of computing resources and ensures smooth completion of data processing tasks through intelligent resource scheduling and task allocation.

[0004] The technical solution to realize the present invention is: a distributed computing resource coordination task scheduling computing platform, including a scheduling coordination module, a job module, a data processing plug-in, a process engine, and middleware.

[0005] The scheduling and coordination module is fixed and has nothing to do with the business. It is designed to be decentralized and highly available. It configures a daemon process and only manages resource and task allocation. It is not responsible for the specific work of task operations and starts, monitors, and shuts down controlled processes.

[0006] The job module is personalized and is divided into access type job, processing type job, output type job and mixed type job modules.

[0007] As a standardized data processing plug-in for the unit organization asset library, the data processing plug-in has a standardized interface to the outside world, and its internal logic is related to the data processing algorithm.

[0008] The middleware includes message middleware and storage middleware. The message middleware mainly implements the publishing, subscription and caching functions of messages, and the storage middleware is NAS devices, shared disks, and distributed object-oriented storage systems.

[0009] The job modules are divided into inseparable jobs and divisible jobs. For inseparable jobs, the job module or the combination of the job module and the data processing plug-in is scheduled as a whole by the scheduling coordination module; for divisible jobs, the data processing of the job module is divided into several links.

[0010] The process engine can create, edit, and save a workflow.

[0011] A distributed computing resource coordination task scheduling calculation method in the field of data processing is implemented by the above-mentioned distributed computing resource coordination task scheduling calculation platform, and the steps are as follows:

[0012] Step 1: Deploy and run the scheduling and coordination module on all computing nodes, and configure the node role as schedulable as needed; according to the leader election rule, each scheduling and coordination module automatically elects a candidate node as the active scheduling node with the help of inconsistent timers and UDP multicast; the scheduling and coordination module on the active scheduling node publishes its own leadership heartbeat to the scheduling and coordination modules on all other candidate scheduling nodes through UDP multicast, and the scheduling and coordination modules of other candidate scheduling nodes establish TCP / IP long connections with the scheduling and coordination modules of the active scheduling node based on the leadership heartbeat; if the scheduling and coordination modules of other candidate scheduling nodes do not receive the leadership heartbeat within the specified time, a new round of leader election is automatically triggered.

[0013] Step 2: The scheduling coordination module on the scheduling node of the activity receives the job / task request from the external system and monitors the resident and scheduled tasks; it performs resource detection within the cluster, and when the resource and time conditions are met, it pulls and automatically starts the job module and data processing plug-in required for the task through the storage middleware, and monitors their status.

[0014] Step 3: For the scenario of providing services to the outside world, the active scheduling node performs resource detection based on the number of service instances to be started, starts the corresponding number of service instances in the cluster, and monitors their status. If the service crashes abnormally, a new service instance is automatically started in time.

[0015] Step 4: For data analysis production scenarios, the activity scheduling node starts the job module. If the job module needs to start the data processing plug-in, it will apply to the activity scheduling node to start and monitor the data processing plug-in. The activity scheduling node creates the corresponding message queue according to the parallelism of the task, and when starting the data processing plug-in, it passes in the startup parameters, which include the task ID, serial number, and the message queue information that needs to be consumed; the job module can subscribe to data, process data, and publish results. It can also use the data processing plug-in to subscribe to data and process data. After publishing the results, the job module subscribes to the plug-in processing results, and after processing, publishes or stores the results.

[0016] Step 5: The job module or data processing plug-in regularly publishes the heartbeat status message and log information of the process. The active scheduling node collects the heartbeat status information of each process and analyzes its status. If the heartbeat status is abnormal, it performs exception processing, logs, and restarts the corresponding job module or data processing plug-in;

[0017] Step 6: When the active scheduling node receives the task end instruction from the external system, it closes the corresponding process through the scheduling coordination module on each computing node and feeds back the task end status to the external system.

[0018] Compared with the prior art, the present invention has the following significant advantages:

[0019] (1) A certain job module can be scheduled to exclusively occupy one computing hardware, or to exclusively occupy multiple computing hardware in a distributed and parallel manner; a certain job module can use part of the computing power of one computing hardware, or to use part of the computing power of multiple computing hardware in a distributed and parallel manner.

[0020] (2) Nodes can automatically join and exit the cluster without restarting or reconfiguration.

[0021] (3) Support resident, scheduled and one-time jobs and tasks.

[0022] (4) The scheduling and coordination module is decentralized, and two or more nodes can achieve high availability of the entire platform.

[0023] (5) Support accepting external requests to start tasks through API or network messages.

[0024] (6) Without hardware virtualization, it can directly support Windows 7 and above operating systems and support cross-platform deployment and operation.

[0025] (7) Can be deployed on a single machine or in a distributed manner. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 This is a timing diagram of distributed computing resource coordination task scheduling computing platform processing of the present invention. DETAILED DESCRIPTION

[0027] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0028] The technical solutions between the various embodiments of the present invention can be combined with each other, but it must be based on the fact that ordinary technicians in the field can implement it. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0029] The following will further introduce the specific implementation method, as well as the technical difficulties and inventive points of this invention in combination with this design example.

[0030] Combination Figure 1 The distributed computing resource coordination task scheduling computing platform described in the present invention includes a scheduling coordination module, a job module, a data processing plug-in, a process engine, and external middleware.

[0031] The scheduling and coordination module is fixed and has nothing to do with the business. It is designed to be decentralized and highly available. It configures a daemon process, which only manages resource and task allocation, but is not responsible for the specific work of the task operation. It starts, monitors and shuts down the controlled process. The job module is personalized and can be divided into access type jobs, processing type jobs, output type jobs and mixed type job modules. The data processing plug-in is a standardized data processing plug-in for the unit organization asset library. It has a standardized interface to the outside world, and its internal logic is related to the data processing algorithm. The middleware includes message middleware and storage middleware. The message middleware can be the common RabbitMQ, RocketMQ, Kafka, etc. on the market, which mainly realizes the publishing, subscription and caching functions of messages. The storage middleware can be NAS devices, shared disks, distributed object-oriented storage systems, etc.

[0032] The job modules can be divided into inseparable jobs and divisible jobs. For inseparable jobs, the job module or the combination of the job module and the data processing plug-in is scheduled as a whole by the scheduling coordination module; for divisible jobs, the data processing of the job module is divided into several links.

[0033] The job module receives the job parameter information from the scheduling and coordination module, and decomposes the task according to the job parameter information. The remote startup of the data processing plug-in is realized by sending a startup application of the data processing plug-in to the scheduling and coordination module. The job module can subscribe to the published results of the data processing plug-in, and assemble and further process them as needed. The job module can monopolize the resources of a computer through multi-threading technology, and multiple computers can process in parallel; it can also share a computer, and multiple processes on multiple computers can process in parallel.

[0034] After the data processing plug-in is started by the scheduling and coordination module, it parses the startup parameters, obtains the task processing information and the subscription and publishing requirements of the message queue, publishes the status heartbeat regularly, executes specific job tasks, and publishes the execution results. The data processing plug-in uses multi-threading technology to monopolize the resources of a computer, and multiple computers process in parallel; it can also share a computer, and multiple computers and multiple processes process in parallel.

[0035] The process engine can create, edit, and save a workflow. There is a docking menu on the canvas to select operators (job modules, data processing plug-ins), and automatically obtain and set parameters by dragging and connecting. The nodes in the workflow are connected according to the arrangement, and the workflow description json data is generated and stored in the database.

[0036] The job module and data processing plug-in can be combined or combined separately through the process engine to form a complete workflow, forming a production line or providing services to the outside world. All jobs, tasks, and services can be monitored in the human-computer interaction interface, and daily scheduling and management can be performed.

[0037] The distributed computing resource coordination task scheduling calculation method in the field of data processing described in the present invention is implemented by the above-mentioned distributed computing resource coordination task scheduling calculation platform, and the steps are as follows:

[0038] Step 1: Deploy and run the scheduling and coordination module on all computing nodes, and configure the node role as schedulable as needed; according to the master election rules, each scheduling and coordination module automatically elects a candidate node as the active scheduling node with the help of inconsistent timers and UDP multicast; the scheduling and coordination module on the active scheduling node publishes its own leadership heartbeat to the scheduling and coordination modules on all other candidate scheduling nodes through UDP multicast, and the scheduling and coordination modules of other candidate scheduling nodes establish a TCP / IP long connection with the scheduling and coordination module of the active scheduling node based on the leadership heartbeat; if the scheduling and coordination modules of other candidate scheduling nodes do not receive the leadership heartbeat within the specified time, a new round of master election is automatically triggered;

[0039] Step 2: The scheduling coordination module on the scheduling node of the activity receives the job / task request from the external system and monitors the resident and scheduled tasks; it detects the resources in the cluster, and when the resources, time and other conditions are met, it pulls and automatically starts the job module and data processing plug-in required for the task through the storage middleware, and monitors their status;

[0040] The specific scheduling rules are as follows:

[0041] Taking into account information from multiple dimensions, prioritize high-priority and computationally intensive tasks. First, calculate the comprehensive priority of task scheduling, which can be calculated based on task priority, CPU usage requirements, memory requirements, GPU requirements, and the weights of each dimension. The calculation formula is as follows:

[0042] Task scheduling comprehensive priority = (task priority * a) + (CPU usage requirement * b) + (memory requirement * c) + e * (GPU requirement * d)

[0043] The task priority is a manually set value ranging from 0 to 1. The higher the task priority, the larger the value. a is the weight for controlling the task priority, and the value of a is 0-100. The CPU usage requirement is a set value ranging from 0 to 1. The higher the CPU usage requirement, the larger the value. When the CPU usage requirement is 1, the task needs to occupy a computing resource exclusively. b is the weight for controlling the CPU usage requirement, and the value of b is 0-100. The memory requirement is a manually set value ranging from 0 to 1. The higher the memory requirement, the larger the value. c is the weight for controlling the memory requirement, and the value of c is 0-100. The GPU requirement is a manually set value ranging from 0 to 1. The higher the task priority, the larger the value. d is the weight for controlling the GPU requirement, and the value of d is 0-100. The value of e is 0 or 1, which is used to distinguish whether GPU resources are needed.

[0044] After calculating the comprehensive priority of task scheduling, the present invention sorts each task according to the comprehensive priority and schedules them in descending order; during resource detection, according to the current load situation, some nodes are avoided from being overloaded and other nodes are idle, and tasks are allocated in order according to the scheduling scores of nodes in the cluster. The specific rules are as follows:

[0045] Node scheduling score = (CPU idle rate * f) + (memory idle rate * g) + h * (GPU idle rate * i) - (node ​​failure rate * j)

[0046] Among them, the value of CPU idle rate is 0 or 1, f is the weight of CPU idle rate, and its value is 0-100; the value of memory idle rate is 0 or 1, g is the weight of memory idle rate, and its value is 0-100; the value of GPU idle rate is 0 or 1, i is the weight of GPU idle rate, and its value is 0-100; node failure rate is the ratio of the number of node failures to the total running time. The higher the node failure rate, the larger the value of j. The value of j is 0-100, which is used to punish nodes with high failure rates.

[0047] Step 3: For the scenario of providing services to the outside world, the active scheduling node performs resource detection based on the number of service instances to be started, starts the corresponding number of service instances in the cluster, and monitors their status. If the service crashes abnormally, a new service instance is automatically started in time.

[0048] Step 4: For data analysis production scenarios, the activity scheduling node starts the job module. If the job module needs to start the data processing plug-in, it applies to the activity scheduling node to start and monitor the data processing plug-in; the activity scheduling node creates the corresponding message queue according to the parallelism of the task, and passes in the startup parameters when starting the data processing plug-in. The parameters include information such as the task ID, serial number, and the message queue that needs to be consumed; the job module can subscribe to data, process the data, and publish the results. It can also use the data processing plug-in to subscribe to the data and process the data. After the results are published, the job module subscribes to the plug-in processing results, and after processing, the results are published or stored.

[0049] Step 5: The job module or data processing plug-in periodically publishes the heartbeat status message and log information of the process. The active scheduling node collects the heartbeat status information of each process and analyzes its status. If the heartbeat status is abnormal, it performs exception handling, logs, and restarts the corresponding job module or data processing plug-in.

[0050] Step 6: When the active scheduling node receives the task end instruction from the external system, it closes the corresponding process through the scheduling coordination module on each computing node and feeds back the task end status to the external system.

[0051] In summary, the present invention can effectively coordinate and manage distributed computing resources and realize efficient scheduling of tasks; at the same time, considering the difficulty of platform deployment, the platform has the characteristics of simple structure, lightweight operation, no complex components, and no installation required.

Claims

1. A distributed computing resource coordination task scheduling computing platform, characterized by: Including scheduling and coordination module, operation module, data processing plug-in, process engine, and middleware; The scheduling and coordination module is fixed and has nothing to do with the business. It is designed to be decentralized and highly available. It configures a daemon process and only manages resource and task allocation. It is not responsible for the specific work of task operations and starts, monitors, and shuts down controlled processes. The job module is personalized and is divided into access type job, processing type job, output type job and mixed type job module; As a standardized data processing plug-in for the unit organization asset library, the data processing plug-in has a standardized interface to the outside world, and its internal logic is related to the data processing algorithm; Middleware includes message middleware and storage middleware. Message middleware mainly implements message publishing, subscription and caching functions, while storage middleware is NAS devices, shared disks and distributed object-oriented storage systems. The job module is divided into inseparable jobs and divisible jobs. For inseparable jobs, the job module or the combination of the job module and the data processing plug-in is scheduled as a whole by the scheduling and coordination module; for divisible jobs, the data processing of the job module is divided into several links; The process engine can create, edit, and save a workflow.

2. The distributed computing resource coordination task scheduling computing platform according to claim 1, characterized in that: The job module receives the job parameter information from the scheduling and coordination module, and performs task decomposition according to the job parameter information; remote startup of the data processing plug-in is realized by sending a startup application of the data processing plug-in to the scheduling and coordination module; The job module subscribes to the published results of the data processing plug-in and assembles and further processes them as needed; The job module can monopolize the resources of a computer through multi-threading technology, and multiple computers can process in parallel; it can also share the use of a computer, and multiple computers can process in parallel with multiple processes; After the data processing plug-in is started by the scheduling and coordination module, it parses the startup parameters to obtain the task processing information and the subscription and publishing requirements of the message queue, publishes the status heartbeat regularly, executes specific job tasks, and publishes the execution results; the data processing plug-in uses multi-threading technology to monopolize the resources of a computer, and multiple computers process in parallel; it can also share a computer, and multiple computers and multiple processes process in parallel.

3. The distributed computing resource coordination task scheduling computing platform according to claim 2 is characterized in that: The process engine can create, edit, and save a workflow. There is a docking menu on the canvas. Select the operator, that is, the job module, and the data processing plug-in. By dragging and connecting, it automatically obtains and sets parameters. The nodes in the workflow are connected according to the arrangement, and the workflow description json data is generated and stored in the database.

4. The distributed computing resource coordination task scheduling computing platform according to claim 3 is characterized by: The job modules and data processing plug-ins are combined or combined separately through the process engine to form a complete workflow, forming a production line or providing services to the outside world; all jobs, tasks, and services can be monitored in the human-computer interaction interface, and daily scheduling and management can be carried out.

5. A distributed computing resource coordination task scheduling calculation method in the field of data processing, characterized in that: The distributed computing resource coordination task scheduling computing platform described in any one of claims 1 to 4 is implemented by the following steps: Step 1: Deploy and run the scheduling and coordination module on all computing nodes, and configure the node role as schedulable as needed; according to the master election rules, each scheduling and coordination module automatically elects a candidate node as the active scheduling node with the help of inconsistent timers and UDP multicast; the scheduling and coordination module on the active scheduling node publishes its own leadership heartbeat to the scheduling and coordination modules on all other candidate scheduling nodes through UDP multicast, and the scheduling and coordination modules of other candidate scheduling nodes establish a TCP / IP long connection with the scheduling and coordination module of the active scheduling node based on the leadership heartbeat; if the scheduling and coordination modules of other candidate scheduling nodes do not receive the leadership heartbeat within the specified time, a new round of master election is automatically triggered; Step 2: The scheduling coordination module on the scheduling node of the activity receives the job / task request from the external system and monitors the resident and scheduled tasks; it detects the resources in the cluster, and when the resource and time conditions are met, it pulls and automatically starts the job module and data processing plug-in required for the task through the storage middleware, and monitors their status; Step 3: For the scenario of providing services to the outside world, the active scheduling node performs resource detection based on the number of service instances to be started, starts the corresponding number of service instances in the cluster, and monitors their status. If the service crashes abnormally, a new service instance is automatically started in time. Step 4: For data analysis production scenarios, the activity scheduling node starts the job module. If the job module needs to start the data processing plug-in, it applies to the activity scheduling node to start and monitor the data processing plug-in; the activity scheduling node creates the corresponding message queue according to the parallelism of the task, and when starting the data processing plug-in, it passes in the startup parameters, which include the task ID, serial number, and the message queue information that needs to be consumed; The job module can subscribe to data, process data, and publish results. It can also use data processing plug-ins to subscribe to data, process data, and publish results. The job module subscribes to the plug-in processing results, and after processing, publishes or stores the results. Step 5: The job module or data processing plug-in regularly publishes the heartbeat status message and log information of the process. The active scheduling node collects the heartbeat status information of each process and analyzes its status. If the heartbeat status is abnormal, it performs exception processing, logs, and restarts the corresponding job module or data processing plug-in; Step 6: When the active scheduling node receives the task end instruction from the external system, it closes the corresponding process through the scheduling coordination module on each computing node and feeds back the task end status to the external system.

6. The distributed computing resource coordination task scheduling calculation method in the data processing field according to claim 5 is characterized in that: In step 2, the scheduling coordination module on the scheduling node of the activity receives the job / task request from the external system and monitors the resident and scheduled tasks; performs resource detection within the cluster, and when the resource and time conditions are met, pulls and automatically starts the job module and data processing plug-in required for the task through the storage middleware, and monitors their status; The specific scheduling rules are as follows: Taking into account information from multiple dimensions, prioritize high-priority and computationally intensive tasks. First, calculate the comprehensive priority of task scheduling, and perform a comprehensive calculation based on task priority, CPU usage requirements, memory requirements, GPU requirements, and the weights of each dimension. The calculation formula is as follows: Task scheduling comprehensive priority = (task priority * a) + (CPU usage requirement * b) + (memory requirement * c) + e * (GPU requirement * d) Among them, the task priority is a manually set value, ranging from 0 to 1. The higher the task priority, the larger the value. a is the weight for controlling the task priority, and the value of a is 0-100; the CPU usage requirement is a set value, ranging from 0 to 1. The higher the CPU usage requirement, the larger the value. When the CPU usage requirement is 1, the task needs to occupy a computing resource exclusively. b is the weight for controlling the CPU usage requirement, and the value of b is 0-100; the memory requirement is a manually set value, ranging from 0 to 1. The higher the memory requirement, the larger the value. c is the weight for controlling the memory requirement, and the value of c is 0-100; the GPU requirement is a manually set value, ranging from 0 to 1. The higher the task priority, the larger the value. d is the weight for controlling the GPU requirement, and the value of d is 0-100. The value of e is 0 or 1, which is used to distinguish whether GPU resources are needed; After calculating the comprehensive priority of task scheduling, sort each task according to the comprehensive priority and schedule them in descending order. During resource detection, according to the current load situation, avoid overloading some nodes and idling other nodes. Tasks are assigned in order according to the scheduling scores of nodes in the cluster. The specific rules are as follows: Node scheduling score = (CPU idle rate * f) + (memory idle rate * g) + h * (GPU idle rate * i) - (node ​​failure rate * j) Among them, the value of CPU idle rate is 0 or 1, f is the weight of CPU idle rate, and its value is 0-100; the value of memory idle rate is 0 or 1, g is the weight of memory idle rate, and its value is 0-100; the value of GPU idle rate is 0 or 1, i is the weight of GPU idle rate, and its value is 0-100; node failure rate is the ratio of the number of node failures to the total running time. The higher the node failure rate, the larger the value of j. The value of j is 0-100, which is used to punish nodes with high failure rates.

Images