Industrial big data computing resource isolation and quantification system

Abstract

The application provides an industrial big data computing resource isolation and quantification system, comprising a resource quantification dynamic integration statistics module, a resource isolation dynamic adjustment module, a DAG flowchart task running resource quantification module, a task creation stage demand resource detection module, a task execution isolation module, a task consumption pressure mechanism module, a high-concurrency scene resource dynamic management and control module, a resource logical allocation layer and physical layer decoupling module, and a resource group dynamic monitoring function module. The application guarantees the stability and reliability of a distributed job scheduling and execution system; effectively solves the scene of reasonable allocation control of distributed job scheduling according to task execution resources and supervision of business demand of cluster resources in the task running process; and can be applied to any application xIn3Plat project in the future.

Description

Technical Field

[0001] This invention relates to the technical field of distributed scheduling systems, specifically to a system for isolating and quantifying industrial big data computing resources, and more particularly to a design method and implementation for isolating and quantifying industrial big data computing resources. Background Technology

[0002] With the advancement of industrial informatization and intelligentization, the digitalization of industrial production and enterprise management processes is continuously deepening. This means that more and more manual operations can be solidified and transformed into individual programs. Based on specified triggering methods, task flow scheduling platforms schedule and execute these tasks at designated times. It is no exaggeration to say that task flow scheduling platforms are one of the core functions of industrial digitalization and intelligentization. This leads to more complex workflow parsing and task scheduling scenarios with fluctuating execution trigger volumes. Specific challenges include:

[0003] 1. Challenges in the regulation of computing resources

[0004] In the practice of promoting industrial informatization, it has been found that many tasks meet the criteria of short individual execution time and low resource consumption, but a large number of tasks are triggered in a short period of time. This requires the scheduling platform to quickly borrow and return resources during task flow execution. However, current industry resource control solutions tend to focus on scenarios where individual tasks are time-consuming and resource-intensive. Therefore, existing resource management solutions cannot meet the current task flow scheduling platform's requirements for resource usage management and performance assurance.

[0005] 2. Computational execution isolation challenges

[0006] Depending on the different task flow scheduling and execution scenarios, some tasks require high real-time performance. For example, timed tasks should not be affected by API call-triggered tasks, or tasks in different workspaces should be isolated based on their workspaces. Therefore, these requirements place high demands on the task flow scheduling platform and present significant challenges in its implementation.

[0007] 3. Challenges of Resource Quantification Calculation in Distributed Workflow Scenarios

[0008] A task flow is composed of steps combined according to different execution dependencies, and these steps can be distinguished as sequential or parallel. Therefore, for a task flow scheduling platform to implement resource management functions, the first requirement is to quantify and statistically analyze execution resources at the task flow granularity. This means performing DAG parsing on the constituent steps of the task flow and calculating the resource usage required for task flow execution using resource optimization algorithms.

[0009] 4. Challenges in Quantifying Distributed Cluster Resources

[0010] The task flow scheduling platform is architected in a distributed and decentralized manner, with the cluster-level architecture supporting horizontal scaling of cluster nodes. However, this highly scalable architecture presents significant challenges in quantifying and statistically analyzing the resources of the entire cluster. This means that while accurately and in real-time quantifying and statistically analyzing the resources of the task flow scheduling cluster, the high scalability of the cluster must also be taken into account.

[0011] 5. Challenges of Dynamic Horizontal Scaling of Distributed Cluster Resources

[0012] For distributed task flow scheduling platforms, it's crucial to ensure high availability and scalability even when horizontally scaling nodes within the cluster. The cluster's resource monitoring capabilities must respond to changes in cluster resources in real time, and resource allocation must be dynamically modified. This ensures the distributed task flow scheduling platform provides high availability and scalability across diverse production environments.

[0013] Based on the above analysis, the distributed task flow scheduling platform, in fulfilling its requirements for resource quantification and isolation, needs to support the following functionalities:

[0014] (1) Quantitative statistics of distributed cluster resources

[0015] (2) Resource supervision supports horizontal dynamic expansion and contraction.

[0016] (3) Supports task flow resource borrowing and returning in high-concurrency scenarios

[0017] (4) Support task execution isolation at the workspace and task flow levels.

[0018] (5) Calculate the optimal runtime resources for task flows consisting of complex execution steps.

[0019] A monitoring device, system, and method for data collection in an industrial environment are disclosed in patent document CN110073301A. The system includes a data collector communicatively connected to multiple input channels and a network architecture; wherein the data collector collects data based on selected data collection routines; the system also includes a data memory structured to store multiple collector routines and collected data, a data collection circuit structured to interpret multiple detection values ​​from the collected data, and a data analysis circuit structured to analyze the collected data and determine the aggregation rate of the data collected from the multiple input channels; if the aggregation rate exceeds the throughput parameters of the network architecture, the data analysis circuit alters the data collection process to reduce the amount of collected data.

[0020] Current distributed task flow scheduling platforms offer high scalability for handling various task types. However, with the continuous increase in daily task submissions in production environments and the unpredictable triggering of API call tasks (e.g., a sudden surge in task submissions within a short period), task consumption backlogs occur. Furthermore, tasks in different workspaces, lacking operational isolation mechanisms, can interfere with each other. These issues will become increasingly apparent with the continued use of task flow scheduling platforms.

[0021] Therefore, based on the above practical and urgent needs, we boldly attempted and took on the challenge, independently developing a function that can dynamically monitor resources of a distributed task flow scheduling cluster in high-concurrency scenarios, while ensuring that the running performance of tasks is not affected. Summary of the Invention

[0022] To address the shortcomings of existing technologies, the purpose of this invention is to provide a system for isolating and quantifying industrial big data computing resources.

[0023] The present invention provides a system for isolating and quantifying industrial big data computing resources, including a resource quantification dynamic integration and statistics module, a resource isolation dynamic adjustment module, a DAG flowchart task operation resource quantification module, a task creation stage demand resource detection module, a task execution isolation module, a task consumption suppression mechanism module, a high-concurrency scenario resource dynamic management and control module, a resource logic allocation layer and physical layer decoupling module, and a resource group dynamic monitoring function module.

[0024] The resource quantification dynamic integration statistics module and the resource isolation dynamic adjustment module work together to trigger the DAG flowchart task execution. The resource quantification module quantifies the resources used by the task flow and obtains the resource usage data, which is then handed over to the task scheduling consumption stage demand resource detection module for resource determination.

[0025] Preferably, the resource quantification and dynamic integration statistics module provides distributed cluster scheduling, computing node resource collection, and quantification and statistical integration display functions.

[0026] Preferably, the resource isolation dynamic adjustment module persists the information of the resource group of the distributed task flow scheduling platform, and dynamically modifies the resource configuration, task execution mode, and performance-related influencing factors of the resource group.

[0027] Preferably, the DAG flowchart task execution resource quantification module parses the constituent steps of the task flow, and quantifies the resources required for the entire task flow during scheduling and execution based on the parsed DAG results and the execution resource optimization algorithm.

[0028] Preferably, the resource requirement detection module in the task creation stage detects whether the resources required for the current task flow can run normally in the bound resource group based on the resource group information corresponding to the workflow.

[0029] Preferably, the task execution isolation module triggers commands for slotted consumption of submitted task flow instances based on the resource group number.

[0030] Preferably, the task consumption suppression mechanism module protects the distributed task flow scheduling cluster from submitting and executing a large number of interface call type task flows.

[0031] Preferably, the high-concurrency scenario resource dynamic management module dynamically adjusts parameters such as resource information, over-allocation weight factor, and task flow consumption mode of the resource group.

[0032] Preferably, the resource logic allocation layer and physical layer decoupling module manages the lifecycle of resource groups and work groups.

[0033] Preferably, the resource group dynamic monitoring function module performs real-time monitoring of the resource usage of the resource groups managed by the entire distributed scheduling platform.

[0034] Compared with the prior art, the present invention has the following beneficial effects:

[0035] 1. This invention ensures the stability and reliability of the distributed job scheduling and execution system;

[0036] 2. This invention effectively solves the business needs of distributed job scheduling to reasonably allocate and control task execution resources, and to monitor cluster resources during task execution.

[0037] 3. This invention can be applied in any future project that uses xIn3Plat. Attached Figure Description

[0038] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0039] Figure 1 This is a flowchart illustrating the principle of the present invention. Detailed Implementation

[0040] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0041] This invention proposes a system for isolating and quantifying industrial big data computing resources, which enables resource monitoring of running tasks in high-concurrency, randomly triggered scenarios by a distributed job scheduling platform, and supports isolated operation of tasks at different granularities.

[0042] This invention mainly includes a resource quantification and dynamic integration statistics module, a resource isolation and dynamic adjustment module, a DAG flowchart task execution resource quantification module, a task creation phase requirement resource detection module, a task execution isolation module, a task consumption suppression mechanism module, a high-concurrency scenario resource dynamic management and control module, a resource logic allocation layer and physical layer decoupling module, and a resource group dynamic monitoring function module.

[0043] The resource quantification and dynamic integration statistics module and the resource isolation and dynamic adjustment module work together in the control layer of the task flow scheduling platform to provide a RESTful interface for configuration and monitoring functions. When a user configures a task flow and the resources used by its steps through the RESTful interface, the DAG flowchart task execution resource quantification module is triggered to quantify the resources used by the task flow and obtain resource usage data. This data is then handed over to the task creation phase resource requirement detection module for resource determination. If successful, the data information is stored in the database; otherwise, the user is prompted with detailed rectification information. After this, the user will trigger the task flow scheduling platform to execute the corresponding task flow through the RESTful interface or a periodic triggering mechanism. This function is handled by the core task execution isolation module and task consumption suppression mechanism module, which schedule and execute the task flow execution commands. In this execution process, the high-concurrency scenario resource dynamic management module implements resource borrowing and returning operations for each task flow instance based on the allocated resource group. Then, specific step instances execute distributed scheduling logic through the resource logic allocation layer and physical layer decoupling module to complete the allocation of step instances to the corresponding computing nodes for computation. The resource usage of the entire task flow scheduling platform is monitored in real time at different granularities by the resource group dynamic monitoring function module.

[0044] The resource quantification and dynamic integration statistics module is responsible for providing distributed cluster scheduling / computing node resource collection, quantification, statistical integration and display functions, such as: real-time resource information monitoring of the entire cluster's resource pool, and real-time resource status monitoring of individual nodes.

[0045] The resource isolation dynamic adjustment module is responsible for persisting information about resource groups in the distributed task flow scheduling platform, as well as dynamically modifying the resource configuration, task execution mode, and performance-related influencing factors of the resource groups.

[0046] The DAG flowchart task execution resource quantification module is responsible for parsing the constituent steps of the task flow during task flow creation. Then, based on the parsed DAG results, it quantifies the resources required for the entire task flow during scheduling and execution using a resource optimization algorithm.

[0047] The task creation phase resource requirement detection module is responsible for detecting, in the final stage of workflow creation, whether the resources required for the current task workflow to run can run normally in the bound resource group, based on the resource group information corresponding to the workflow.

[0048] The task execution isolation module is responsible for processing task flow instances at the task sub-scheduling points of the distributed scheduling cluster. Based on the resource group number, it triggers commands for slotted consumption of the submitted task flow instances, thereby achieving workspace-level task flow scheduling and execution isolation.

[0049] The task consumption suppression mechanism module is responsible for implementing the task flow scheduling and execution suppression mechanism at the resource group and task granularity. This mechanism protects the distributed task flow scheduling cluster from task flow backlog caused by a large number of interface call types being submitted for scheduling and execution in a very short time, which would affect the task flow consumption of the entire platform.

[0050] The high-concurrency scenario resource dynamic management module is responsible for dynamically adjusting parameters such as resource information, over-provisioning weight factor, and task flow consumption mode of resource groups.

[0051] The resource logical allocation layer and physical layer decoupling module is responsible for the lifecycle management of resource groups and workgroups, as well as decoupling the binding relationship between actual computing nodes and resource groups and making their use transparent. For example, it handles the creation, updating, and deletion of resource groups / workgroups.

[0052] The resource group dynamic monitoring module is responsible for real-time monitoring of the resource usage of the resource groups managed by the entire distributed scheduling platform. For example, it can monitor the resource usage of a certain resource group and the current task queuing status.

[0053] In the module decoupling the resource logic allocation layer from the physical layer, a RESTful interface is implemented, and a web-based visual configuration page is provided based on this interface. This function allows the creation of resource groups and workgroups, and the allocation of appropriate resources from the task flow cluster's resource pool to these resource groups according to suitable use cases.

[0054] In the workspace configuration settings, select appropriate resource groups based on the usage scenario and configure them to the current workspace to achieve workspace-level task flow scheduling and execution isolation, as well as a resource monitoring mechanism.

[0055] In the DAG flowchart task execution resource quantification module, after the steps of the task flow are parsed according to the DAG, the optimal resources required for the execution of the task flow in the current workspace are calculated by the optimal execution resource algorithm, and the obtained information is persisted to improve scheduling execution performance.

[0056] In the high-concurrency scenario resource dynamic management module, a TCP access persistent storage layer function is provided. Based on the resource group number and the current resource status of the resource group in the polled task flow execution command, the atomic mechanism of the DB layer ensures the consistency of the corresponding resource requests during task flow execution.

[0057] In the task consumption suppression mechanism, the task flow queue information (number of task flows executed + number of task flows queued) of the resource group to which the task flow belongs is used to calculate and analyze the data to determine whether the current resource group is in a suppressed state. If the state is normal, the task flow scheduling logic continues to be executed; otherwise, the task flow executes the scheduling logic to roll back and wait for the next task consumption. At the same time, the task flow scheduling submission interface is in a temporary circuit breaker protection mode.

[0058] In the task execution isolation module, the task queue in each resource group is consumed fairly by the slotted consumption mechanism at the workspace or task flow level, and the calculation steps are sent to the corresponding actual physical nodes according to the corresponding work group, thereby realizing the scheduling and execution isolation function of task flow instances.

[0059] This invention ensures the stability and reliability of distributed job scheduling and execution systems; it effectively solves the business needs of distributed job scheduling to reasonably allocate and control task execution resources, and to monitor cluster resources during task execution; this invention can be applied in any future project using xIn3Plat.

[0060] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0061] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A system for isolating and quantifying industrial big data computing resources, characterized in that, It includes a resource quantification and dynamic integration statistics module, a resource isolation and dynamic adjustment module, a DAG flowchart task execution resource quantification module, a task creation phase requirement resource detection module, a task execution isolation module, a task consumption suppression mechanism module, a high-concurrency scenario resource dynamic management and control module, a resource logic allocation layer and physical layer decoupling module, and a resource group dynamic monitoring function module. The resource quantification dynamic integration and statistics module and the resource isolation dynamic adjustment module work together to trigger the DAG flowchart task execution resource quantification module to quantify the resources used by the task flow and obtain the resource usage data, which is then handed over to the task creation phase demand resource detection module for resource determination. The resource isolation dynamic adjustment module persists information about resource groups in the distributed task flow scheduling platform and dynamically modifies the resource configuration, task execution mode, and performance-related influencing factors of the resource groups. The DAG flowchart task execution resource quantification module parses the constituent steps of the task flow, and based on the parsed DAG results, quantifies the resources required for the entire task flow during scheduling and execution using the execution resource optimization algorithm. The resource requirement detection module in the task creation phase detects whether the resources required for the current task flow to run can run normally in the bound resource group based on the resource group information corresponding to the workflow. The resource logic allocation layer and physical layer decoupling module is responsible for the lifecycle management of resource groups and work groups, as well as decoupling the binding relationship between actual computing nodes and resource groups and making their use transparent. In the module for decoupling the resource logic allocation layer from the physical layer, a RESTful interface is implemented, and a web-based visual configuration page is provided based on this interface. This function is used to create resource groups and work groups, and appropriate resources are allocated from the resource pool of the task flow cluster to the resource groups according to the appropriate use case. In the high-concurrency scenario resource dynamic management module, a TCP access persistent storage layer function is provided. Based on the resource group number and the current resource status of the resource group in the polled task flow execution command, the atomic mechanism of the DB layer ensures the consistency of the corresponding resource requests during task flow execution.

2. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The resource quantification and dynamic integration statistics module provides distributed cluster scheduling, computing node resource collection, and quantification and statistical integration display functions.

3. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The task execution isolation module triggers commands to consume submitted task flow instances in slots based on the resource group number.

4. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The task consumption suppression mechanism module protects the distributed task flow scheduling cluster from submitting and executing a large number of interface call type task flows.

5. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The high-concurrency scenario resource dynamic management module dynamically adjusts the resource information, over-allocation weight factor, and task flow consumption mode parameters of the resource group.

6. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The resource logic allocation layer and physical layer decoupling module manages the lifecycle of resource groups and work groups.

7. The system for isolating and quantifying industrial big data computing resources according to claim 1, characterized in that, The resource group dynamic monitoring module provides real-time monitoring of the resource usage of the resource groups managed by the entire distributed scheduling platform.

Images